How do you think about probability? Many people first learn about it as a long-term frequency. If you flip a fair coin 10,000 times, you expect to get heads about 5,000 times. The probability of heads, 50%, is the proportion you'd see over countless trials.

This is the core of the frequentist approach to statistics. It defines probability as the relative frequency of an event in the long run. It's objective and works well for repeatable events like coin flips or dice rolls.

But what about questions where you can't run thousands of trials? For example, what's the probability that a specific political candidate will win the next election? Or the probability that a newly discovered planet harbors life? These events will only happen once. A frequentist approach doesn't quite fit.

Bayesian statistics offers a different perspective. It treats probability as a degree of belief or confidence in a statement. This belief can be updated as new evidence becomes available. It's a measure of our uncertainty about the world.

The process of updating beliefs in a Bayesian framework has three key ingredients: the prior, the likelihood, and the posterior.

Imagine you're trying to determine the average weight of a house cat. You haven't weighed any cats yet, but you know they aren't as light as a hamster or as heavy as a lion. Your prior distribution might be centered around 10 pounds, with a wide spread to reflect your uncertainty.

Now, you go out and weigh a sample of 20 cats. This is your data. The likelihood function uses this data to evaluate your initial belief. If your sample of cats has an average weight of 12 pounds, the likelihood will peak at 12. It tells you that, given the data, a 12-pound average is more plausible than a 10-pound or 15-pound average.

The posterior is the result of the Bayesian process. You combine your prior belief (centered at 10 pounds) with the evidence from your data (the likelihood, centered at 12 pounds). The resulting posterior distribution will be a new belief, likely centered somewhere between 10 and 12 pounds. It will also be narrower than your prior, reflecting that you are now less uncertain.

This whole process is formalized by a simple but powerful equation called Bayes' Theorem.

Let's use $H$ to represent a hypothesis (e.g., the average cat weight is 10 lbs) and $D$ to represent our data (the 20 cat weights we collected). Bayes' Theorem looks like this:

Let's break that down into the concepts we just learned:

In words, the theorem states:

The key takeaway is that Bayesian statistics provides a formal framework for updating what we know. We start with a prior belief, collect data, and use that data to arrive at a more informed posterior belief. It's a natural model for how we learn about the world.