In machine learning, we often talk about training as a process of optimization. We define a loss function and use techniques like gradient descent to find the model parameters that minimize it. This is a powerful and useful framing, but it's not the only one. A Bayesian perspective reframes training not as a hunt for a single best set of parameters, but as a process of refining our beliefs about them. Instead of seeking one point, we shape a whole landscape of possibilities.

We start with a prior distribution, which represents our initial beliefs about the model's parameters before we've seen any data. This could be a very broad, uncertain belief. Then, we introduce evidence in the form of training data. Using Bayes' theorem, we combine our prior belief with the evidence from the data to form an updated, more refined belief called the posterior distribution. The posterior is narrower and more peaked, representing reduced uncertainty.

This process isn’t a one-time event. As new data arrives, the model can update its beliefs recursively. The posterior distribution calculated from the first batch of data becomes the prior for the second batch. This iterative refinement is the heart of Bayesian learning. It’s how a system can continuously adapt and reduce its uncertainty over time.

The key is the likelihood function, which connects our model to the data. It answers the question: "Given my current belief about the model's parameters, how likely is the data I just observed?" For diagnostic data like error rates or performance metrics, the likelihood function translates those raw numbers into a formal piece of evidence that can be used to mathematically update the model's beliefs. Each update compresses the uncertainty, pushing the posterior distribution towards regions of high parameter plausibility.

In practice, calculating the posterior distribution directly can be mathematically messy or even impossible. This is where a clever mathematical trick comes in handy: using conjugate priors with our likelihood function. When a prior distribution is "conjugate" to a likelihood function, the resulting posterior distribution belongs to the same family of distributions as the prior. For example, if you have a binomial likelihood and you choose a beta distribution as your prior, the posterior will also be a beta distribution. This simplifies the math tremendously. Instead of re-calculating a complex distribution from scratch, an update simply involves changing the parameters of the existing distribution, like updating the $\alpha$ and $\beta$ parameters of a Beta distribution. This makes recursive updates computationally feasible.

When direct calculation of the posterior is too complex, as is often the case in deep learning, we need another approach. Variational inference is a common technique that approximates the true posterior $P(\theta|D)$ with a simpler, tractable distribution $Q(\theta)$. But how do we know if our approximation is any good? We need a metric to guide us.

This is the role of the Evidence Lower Bound (ELBO). Maximizing the ELBO is equivalent to minimizing the Kullback-Leibler (KL) divergence between our approximation $Q(\theta)$ and the true posterior $P(\theta|D)$. In simpler terms, making the ELBO as large as possible makes our approximation as close as possible to the real posterior.

The ELBO beautifully connects optimization back to our Bayesian framework. It gives us a concrete objective function to maximize. By doing so, we are performing uncertainty compression, squeezing our approximate belief $Q(\theta)$ to be as accurate a representation of the true posterior as possible, guided by the evidence in our data.