In linear regression, we often start with a familiar goal: find the single straight line that best fits our data. This method, known as Ordinary Least Squares (OLS), gives us one set of coefficients—one slope and one intercept—that minimize the squared distance between the line and the data points. This is a point estimate. It's clean, simple, and often very effective. But it has a crucial limitation: it communicates no uncertainty. It gives us the 'best' answer without telling us how confident we should be in that answer.

What if, instead of asking for the one best line, we asked for a probability distribution over all possible lines? This shift in perspective is the core of Bayesian linear regression. We move from finding a single point to describing a landscape of possibilities. Instead of one answer, we get a range of plausible answers, each with a probability attached. This approach acknowledges that our data is finite and noisy, so our estimate of the model's parameters should reflect that uncertainty.

This probabilistic viewpoint requires us to think about two key ingredients: the likelihood and the prior.

The likelihood function asks: given a particular set of weights (a specific line), what is the probability of observing our actual data? For linear regression, we typically assume the errors, or residuals, are drawn from a Gaussian (normal) distribution with a mean of zero. Therefore, the likelihood is the product of the probabilities of each data point, assuming they were generated by a line plus some Gaussian noise.

The frequentist approach of Maximum Likelihood Estimation (MLE) stops here. It finds the parameters $\beta$ that maximize this likelihood function. It turns out that for linear regression, maximizing this likelihood is equivalent to minimizing the sum of squared errors—giving you the same result as OLS.

The Bayesian approach, pioneered by Thomas Bayes, introduces the second ingredient: the prior distribution, $P(\beta)$. The prior is a probability distribution that represents our beliefs about the parameters before we see any data. It's where we can incorporate domain knowledge or a preference for simpler models. For example, we might start with a prior belief that the regression weights are likely to be small and centered around zero.

By combining the prior with the likelihood using Bayes' theorem, we get the posterior distribution, $P(\beta | y, X)$.

Since the denominator, $P(y|X)$, doesn't depend on $\beta$, we often work with the proportional relationship:

This brings us to the core difference in estimation. The frequentist approach uses Maximum Likelihood Estimation (MLE), while a common Bayesian estimation method is Maximum A Posteriori (MAP).

Notice that MAP is like MLE with an extra term: the prior. This prior acts as a regularizer. For example, if we use a Gaussian prior for our weights centered at zero, the MAP estimate becomes equivalent to Ridge Regression. The prior effectively penalizes large weights, helping to prevent overfitting.

While MAP provides a single point estimate, the true power of the Bayesian approach lies in using the entire posterior distribution. Instead of just one line, we have a whole family of lines, which gives us a credible interval for our predictions and a richer understanding of the model's uncertainty.

This shift from point estimates to distributions is a fundamental change in how we think about modeling. It forces us to be explicit about our assumptions and provides a complete picture of what our model has learned from the data.