Bayesian Inference

Bayes’ Rule

Bayes’ Rule is a wonderful piece of math that every decision-maker should have a conceptual grasp of at the least. It is the heart of subjective probability and essentially defines a method of updating our beliefs in light of new evidence. Every one of us uses it in our daily lives without ever knowing it and it looks like this:

\[P(A|B) = \frac{P(B|A)\cdot P(A)}{P(B)}\]
\[\text{Posterior} = \frac{\text{Likelihood } \cdot \text{ Prior}}{\text{Evidence}}\]

In plain English, Bayes’ Rule tells us how strong our belief is in \(A\) after observing new evidence \(B\). To go more in depth let us define some more terms.

  • prior - our prior or initial belief that \(A\) is true

  • evidence - the total probability of observing \(B\) in an experiment

  • likelihood - the probability of observing \(B\) if \(A\) is in fact true

  • posterior - our belief in \(A\) after considering the evidence

A Young Example

As a story example, imagine a middle school student who is trying to make friends with another popular student. They may not feel comfortable asking right out if the popular one considers them a friend, but they can gather information from their interactions at school. They honestly believe there is a chance that the popular student considers them a friend.

Let’s say the popular student is going to have a party in the near future. One indicator that they consider our protagonist to be a friend would be if they invite them to this party. If they do consider them a friend it is likely but not guaranteed that they would invite them to this party. Perhaps only family or their closest friends are invited, or perhaps they invite everyone in their class. In contrast, even if the popular student does not consider them to be friends, they may still invite the protagonist to the party. In the case that they are invited to the party, the likelihood and the evidence are both relatively large so the protagonist’s posterior belief could be similar to their prior belief.

Now what if the popular student were to invite them to sit together at lunch? First, our protagonist would find this event nearly impossible. However, it is much more probable if they really are friends. With a likelihood that is much larger in proportion to the evidence, our protagonist’s posterior belief would be much stronger than their prior belief.

A Discrete Case

[ ]: