Introduction to Maximum Likelihood Estimation

What's the Best Guess?

Imagine you find a strange coin. You flip it 10 times and get this sequence: Heads, Tails, Heads, Heads, Heads, Tails, Heads, Heads, Tails, Heads. That’s 7 heads and 3 tails. What's your best guess for the true probability of this coin landing on heads? Most people would say 7/10 or 0.7. This intuition is the heart of a powerful statistical method called Maximum Likelihood Estimation (MLE).

Maximum likelihood estimation is a method that determines values for the parameters of a model.

MLE asks a simple question: out of all possible values for a parameter (like the probability of heads), which value makes the data we actually observed the most probable? It's a way of fitting a model to data by finding the parameters that best explain what we've seen.

The Likelihood Function

To find this "best" parameter, we use something called a likelihood function. This function, written as $L(\theta | \text{data})$ , calculates the probability of observing our specific data, given a particular value for the parameter $\theta$ . Our goal is to find the value of $\theta$ that maximizes this function.

For a series of independent events (like our coin flips), the total likelihood is the product of the individual probabilities:

L(\theta | x_1, ..., x_n) = \prod_{i=1}^{n} P(x_i | \theta)

Working with products can be tricky, especially when using calculus to find a maximum. So, we make our lives easier by using the log-likelihood function. The logarithm turns products into sums, which are much simpler to work with.

Since the logarithm function is monotonically increasing, the parameter value that maximizes the likelihood is the same one that maximizes the log-likelihood.

The log-likelihood, often written as $\ell(\theta | \text{data})$ or $\ln L$ , is just the natural log of the likelihood function:

\ell(\theta | x_1, ..., x_n) = \ln \left( \prod_{i=1}^{n} P(x_i | \theta) \right) = \sum_{i=1}^{n} \ln P(x_i | \theta)

Now, we have a function we can maximize using calculus: take the derivative with respect to the parameter, set it to zero, and solve.

Example: A Biased Coin

Let's formalize our coin flip example. A single coin flip follows a Bernoulli distribution, which has one parameter, $p$ , the probability of heads (a "success"). Let's say getting heads is $x=1$ and tails is $x=0$ . The probability of any single outcome is:

P(x|p) = p^x (1-p)^{1-x}

Suppose we observe $n$ flips with $k$ heads. Our data consists of $k$ ones and $n-k$ zeros. The log-likelihood function for all our data is the sum of the individual log-probabilities:

\begin{aligned} \ell(p | \text{data}) &= \sum_{i=1}^{n} \ln \left( p^{x_i} (1-p)^{1-x_i} \right) \\ &= \sum_{i=1}^{n} \left( x_i \ln(p) + (1-x_i) \ln(1-p) \right) \\ &= k \ln(p) + (n-k) \ln(1-p) \end{aligned}

To find the value of $p$ that maximizes this, we take the derivative with respect to $p$ and set it to zero.

\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0

Solving this equation for $p$ gives us the maximum likelihood estimate, which we denote with a hat: $\hat{p}$ .

\begin{aligned} \frac{k}{p} &= \frac{n-k}{1-p} \\ k(1-p) &= p(n-k) \\ k - kp &= np - kp \\ k &= np \\ \hat{p} &= \frac{k}{n} \end{aligned}

The result is exactly what our intuition told us! The best estimate for the probability of heads is the number of heads we observed divided by the total number of flips. For our initial example, $\hat{p} = 7/10 = 0.7$ .

Example: A Normal Distribution

MLE isn't just for simple cases. Let's find the MLE for the parameters of a normal (Gaussian) distribution: the mean $\mu$ and the variance $\sigma^2$ . The probability density function (PDF) for a single data point $x_i$ from a normal distribution is:

f(x_i | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)

The log-likelihood for $n$ data points is the sum of the logs of their PDFs:

\begin{aligned} \ell(\mu, \sigma^2 | \text{data}) &= \sum_{i=1}^{n} \ln \left( \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right) \right) \\ &= \sum_{i=1}^{n} \left( -\frac{1}{2}\ln(2\pi\sigma^2) - \frac{(x_i - \mu)^2}{2\sigma^2} \right) \\ &= -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2 \end{aligned}

We have two parameters, so we need to take a partial derivative with respect to each one and set them to zero.

First, for the mean $\mu$ :

\begin{aligned} \frac{\partial\ell}{\partial\mu} &= -\frac{1}{2\sigma^2} \sum_{i=1}^{n} 2(x_i - \mu)(-1) = 0 \\ \frac{1}{\sigma^2} \sum_{i=1}^{n} (x_i - \mu) &= 0 \\ \sum x_i - n\mu &= 0 \\ \hat{\mu} &= \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x} \end{aligned}

The MLE for the population mean is simply the sample mean. Again, this matches our intuition.

Next, for the variance $\sigma^2$ :

\begin{aligned} \frac{\partial\ell}{\partial\sigma^2} &= -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2} \sum_{i=1}^{n}(x_i - \mu)^2 = 0 \\ \frac{1}{2(\sigma^2)^2} \sum (x_i - \mu)^2 &= \frac{n}{2\sigma^2} \\ \hat{\sigma}^2 &= \frac{1}{n}\sum_{i=1}^{n}(x_i - \hat{\mu})^2 \end{aligned}

The MLE for the variance is the average of the squared differences from the sample mean. It provides a formal justification for using common sense estimates for the parameters of a distribution.

Let's review what we've learned before you try a few problems.

Now, let's test your understanding.

Quiz Questions 1/5

What is the primary goal of Maximum Likelihood Estimation (MLE)?

Quiz Questions 2/5

In the context of MLE, why is the log-likelihood function often used instead of the likelihood function itself?

Maximum Likelihood Estimation provides a consistent and powerful framework for estimating model parameters. By finding the values that make our observed data most probable, it turns an intuitive idea into a rigorous mathematical tool.

Mastering Maximum Likelihood Estimation