Bayesian Thinking Explained

We are all Bayesians, even if we don't know what that means

We are all Bayesians—or at least we should be—in that we use new information to adjust our beliefs. That government official you think is crooked? New evidence seems to exonerate him, and so you adjust your assessment of his crookedness to match. You form a hypothesis about him based on the information you have. Then you get new information, which causes you to adjust your hypothesis. You might still think he’s crooked, but now you’re less sure.

What is Bayesian thinking? How can we explain it to others?

I recently saw someone perform a simple experiment to demonstrate how Bayesian thinking works. It made a lot of sense to me. I hope you like it, too.

In this experiment, you sit behind a screen and I roll a fair, six-sided die. Your job is to use what I tell you to guess whether the first roll of the die was a 1, 2, 3, 4, 5, or 6. However, I won’t directly tell you anything about the first roll. What I’ll tell you is whether the second, third, fourth, etc. rolls are numerically the same as the first roll, numerically less than the first roll, or numerically greater than the first roll. Those are the only three possibilities.

For example, if the first roll was a 6 and the second roll was a 4, I would tell you the second roll was numerically less than the first roll.

With no information from me, here is what you should assign as the probabilities for that first die roll. Based on this limited amount of information, this is your hypothesis, which in the Bayesian way of thinking, is called your “priors.”

I did this experiment today with a real die. What follows are the results of the next five rolls. After rolling, I provide the following information to you:

Let’s think about the results.

Given that there were two rolls less than the first roll, the first roll couldn’t have been a 1. Given that there was one roll greater than the first roll, the first roll couldn’t have been a 6. The probability of the first roll being a 1 or a 6 is 0%.

We can calculate the probabilities for 2, 3, 4, and 5, but let’s think about this a little. If the first roll was a 2, rolls three and five must have been a 1. That’s not very likely. If the first roll was a 3, rolls three and five could have been a 1 or a 2. That’s more likely. If the first roll was a 4, rolls three and five could have been a 1, 2, or a 3. That’s even more likely. A first roll of a 5 could have had a 1, 2, 3, or 4 for rolls three and five, which is even more likely.

But we must remember that one of the rolls was greater than the first roll. If the first roll was a 5, the sixth roll could only have been a 6, while if the first roll was a 4, the sixth roll could have been a 5 or a 6, which is more likely. If the first roll was a 3, the sixth roll could have been a 4, 5, or 6, while if the first roll was a 2, the sixth roll could have been a 3, 4, 5, or 6.

Based on what we know, the first roll was probably a 3, 4, or 5.

Here are the calculated probabilities based on five rolls (I didn’t show the mathematics to minimize complexity):

Now let’s do five more rolls to get more information. Here are the results:

Here are the new probabilities, which, importantly, include information from all the rolls, not just the previous five rolls:

Notice how much the probability for 5 dropped with the new information. As we add information, we adjust our priors. Our priors include everything up to the current point in time. Here are the newest die rolls:

Here are the latest probabilities, which, again, include information from all the rolls:

Let’s do one last set of rolls of the die. Here is the information:

Here are the “final” probabilities for this experiment, which include information from rolls 2-21:

Any guess what the first roll was? The first roll was a 4. Our priors placed the probability for a roll of 4 at 16.7%. With our first set of information, we calculated a probability of 36% for 4. If we had to guess at that point, we would have guessed 4 and been correct. With the inclusion of the second set, the probability went to 42%. With the inclusion of the third set, the probability went to 55%. Finally, we ended with a probability of 70%.

With enough rolls of the die, the probability of the first roll being a 4 would approach 100%.

The important take-away from this experiment is that we start with some information, called our priors, and that we use new information to adjust our prior beliefs. Now, those beliefs become our priors, and we can add yet more information. This can go and on. The experiment shown here lays out the process for this and the specific procedure—had we shown the math—for adjusting our priors to include new information.

Now we can all be Bayesians and know what that means.