How does a model 'learn'? At its core, it's a process of finding the best set of internal parameters to explain the data it's given. Think of it like tuning a radio. You twist the dial (the parameters) until the signal (the data) comes in as clearly as possible. Statistical estimation frameworks provide the mathematical rules for this tuning process. Two of the most important frameworks are Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) estimation.

The first step is understanding the concept of a likelihood function a concept central to both MLE and MAP. Unlike a probability function, which tells you the chance of seeing a certain outcome given fixed parameters, a likelihood function does the reverse. It takes your observed data as fixed and asks: under which set of parameters is this data most likely to have occurred?

Maximum Likelihood Estimation, or MLE, takes a simple and intuitive approach. It says the best parameters for our model are the ones that maximize the likelihood function. In other words, let's choose the parameter values that make our observed data the most probable outcome. It's a purely data-driven approach, making no prior assumptions about what the parameters should be.

For our coin flip example, MLE would conclude that the best estimate for the probability of heads is 7/10, or 0.7. This makes sense; it's the value that directly reflects the data. However, MLE's total reliance on data has a downside. If you only flip the coin 3 times and get 3 heads, MLE would estimate the probability of heads as 1.0. This is almost certainly wrong and is a classic example of overfitting to a small dataset.

What if we have some prior knowledge? We know most coins are roughly fair. It would be useful to incorporate this belief into our estimation. This is the core idea behind Bayesian inference, which provides the foundation for MAP estimation. It combines our initial belief, called the prior distribution, with the evidence from our data (the likelihood) to form an updated belief, called the posterior distribution.

Maximum A Posteriori (MAP) estimation simply finds the peak of this posterior distribution. It answers the question: what parameter value is most probable after seeing the data and considering our prior belief? This balances what the data is telling us with what we already thought to be true.

Notice the difference? MAP includes the prior term $P(\theta)$. If our prior belief is that the coin is fair, a MAP estimate for 3 heads in 3 flips will be pulled away from 1.0 and back towards 0.5. The prior acts as a guardrail against extreme conclusions drawn from limited data.

This brings us to a crucial connection in machine learning. The act of adding a prior in MAP estimation is mathematically equivalent to adding a regularization term to a loss function. Regularization is a technique used to prevent overfitting by penalizing complex models.

This insight recasts model training in a new light. Minimizing squared error in linear regression is an MLE problem, assuming the errors are normally distributed. When you add an L2 penalty term to that loss function, you are no longer doing MLE. You are now performing MAP estimation, with an implicit prior belief that the model's weights should be small and centered around zero.

The final row of the table is key. The influence of the prior diminishes as you collect more data. With an enormous dataset, the likelihood term overwhelms the prior, and the MAP estimate becomes virtually identical to the MLE estimate. In this sense, MAP is most useful when data is scarce, and our prior knowledge can help guide the model to a more sensible solution.

Understanding these frameworks moves you from just fitting lines to data to making principled choices about how your models learn. It clarifies why we use certain loss functions and how techniques like regularization are not just ad-hoc tricks, but are deeply rooted in statistical theory.