How do you form opinions or make decisions? You probably start with some initial belief, and then you adjust that belief as you encounter new information. If you think a restaurant is good based on a friend's recommendation, you might change your mind after reading several bad reviews. This process of updating your beliefs with new evidence is the core idea behind Bayesian statistics.

This might sound like simple common sense, but it marks a significant departure from another major school of thought in statistics: frequentism. A frequentist approach generally avoids incorporating prior beliefs. It focuses strictly on the data collected from a specific experiment to draw conclusions. For a frequentist, probability is about the long-run frequency of an event. If you flip a coin 100 times and it lands on heads 55 times, a frequentist would say the probability of heads is 0.55 based on that data.

A Bayesian, on the other hand, would start with a prior belief—perhaps that the coin is fair, with a 50% chance of heads. The 100 coin flips are new evidence. This evidence is then used to update the initial belief, resulting in a posterior belief that is a blend of the prior and the evidence. The final belief would be somewhere between 50% and 55%.

This process of updating beliefs is formalized by a simple but powerful equation called Bayes' theorem. It tells us exactly how to combine our prior beliefs with new evidence.

The theorem looks like this:

Let's break down what each part means in the context of our belief-updating framework. Here, 'A' is our belief or hypothesis, and 'B' is the new evidence we've observed.

In essence, the theorem says our updated belief (Posterior) is proportional to our initial belief (Prior) multiplied by how well that belief explains the new data (Likelihood).

Let's use a classic example to see this in action. Imagine a rare disease that affects 1 in every 10,000 people. There's a test for this disease that is 99% accurate. This means it correctly identifies 99% of people who have the disease (true positive) and correctly identifies 99% of people who don't (true negative).

You take the test and it comes back positive. What is the probability that you actually have the disease?

Our intuition might say it's 99%, but let's apply Bayes' theorem.

Our hypothesis (A) is: "You have the disease." Our evidence (B) is: "You tested positive."

Let's identify our components:

Now we plug everything into Bayes' theorem to find the posterior probability, $P(A|B)$.

The result is about 0.98%, or just under 1%. Even with a positive result from a 99% accurate test, the chance you have the disease is still very low. Why? Because the disease itself is so rare. The prior belief has a huge impact on the final outcome. The vast majority of positive tests will be false positives from the large pool of healthy people.

This kind of thinking is powerful. It forces us to confront our biases and update our beliefs in a structured way, which is crucial in fields from medicine and law to machine learning and finance.

Now, let's test your understanding of these core concepts.

By starting with a prior and updating it with evidence, Bayesian inference provides a flexible and intuitive framework for reasoning under uncertainty.