So far, we've treated probability as a long-run frequency. If you flip a coin enough times, the proportion of heads will approach 50%. This is the frequentist perspective: probability is an objective feature of the world, and parameters like the true mean of a population are fixed, unknown constants.

Bayesian statistics starts from a different place. It defines probability as a reasonable expectation or a 'degree of belief'. Instead of one single, true value for a parameter, we have a whole distribution of possible values, each with its own likelihood.

This doesn't mean Bayesian probability is arbitrary. It's a structured way to quantify uncertainty. We start with an initial belief (a 'prior') and, as we collect data, we update that belief. It's a formal process for learning from evidence.

The most significant shift from frequentist to Bayesian thinking is how we treat parameters. In the frequentist world, a parameter—like the average height of a population—is a single, fixed number. We may not know it, but it exists. Our goal is to estimate it.

In the Bayesian world, that same parameter is a random variable. This is a crucial distinction. Because it's a random variable, it doesn't have one value; it has a probability distribution. This distribution reflects our uncertainty about the parameter. Where the distribution is high, we have a strong belief the parameter's value is there. Where it's low, we believe it's unlikely to be.

This approach allows us to make direct probability statements about parameters. For example, we can say "There is a 95% probability that the true mean is between 10 and 20." This is something a frequentist confidence interval, despite its name, doesn't allow you to say. It's a more intuitive way to talk about what we've learned from our data.

Frequentist hypothesis testing, which relies on p-values, answers a specific question: "Assuming the null hypothesis is true, what is the probability of observing data this extreme?" This can be a bit convoluted. We get the probability of the data, assuming our hypothesis ($P(\text{Data} | \text{Hypothesis})$).

But often, what we really want to know is the reverse. We have the data—that's the evidence we've collected. We want to know the probability that our hypothesis is true, given this evidence. We want $P(\text{Hypothesis} | \text{Data})$.

This is called inverse probability, and it's the heart of the Bayesian approach. It allows us to flip the conditional probability around to answer the question we're usually more interested in. This is accomplished using Bayes' Theorem, which we'll explore next. It's the engine that updates our beliefs, taking our prior knowledge and combining it with new evidence to form an updated, more informed belief.

Ready to test your understanding of this conceptual shift?

This shift in perspective—from probability as frequency to probability as belief—opens up a powerful and intuitive way to reason about uncertainty. Next, we'll dive into the mechanics of how it works.