In machine learning, we often try to estimate unknown parameters of a population from a sample of data. This process is called point estimation. We might want to estimate the average height of all men in the UK, or the failure rate of a specific component in a jet engine. Our estimate, called an estimator, is a function of the data. But how do we know if our estimator is any good? We judge it based on a few key properties.

Think of an archer aiming at a target. If their arrows cluster around the bullseye on average, they are unbiased. If the arrows are tightly grouped, they are precise (or have low variance). The ideal archer is both unbiased and precise, consistently hitting the bullseye.

An estimator is unbiased if its expected value is equal to the true value of the parameter it's trying to estimate. In other words, if you were to repeat your sampling process many times, the average of your estimates would be spot on. Mathematically, for an estimator $\hat{\theta}$ of a parameter $\theta$, we want $E[\hat{\theta}] = \theta$. Any systematic deviation from this is called bias.

Next, an estimator is consistent if it gets closer to the true parameter value as the sample size increases. This is a crucial property. It means that collecting more data will improve our estimate, which is what we'd intuitively expect. A consistent estimator might be slightly biased with a small dataset, but that bias disappears as we gather more information.

Finally, we have efficiency. If we have two different unbiased estimators for the same parameter, the one with the smaller variance is more efficient. It gives more precise estimates. An efficient estimator makes the most out of the data you have, providing the tightest possible range of likely values for the parameter.

So how do we find estimators that have these desirable properties? One of the most powerful and widely used methods is Maximum Likelihood Estimation (MLE). The core idea is simple but profound: find the parameter values that make the observed data most probable. In other words, given our data, what parameter values 'maximise the likelihood' of us seeing this exact dataset?

Let's say we have a set of data points $x_1, x_2, ..., x_n$, which we assume are drawn independently from a probability distribution $f(x | \theta)$ with an unknown parameter $\theta$. The likelihood function is the product of the probabilities of each individual data point:

Working with products can be mathematically tricky. It's much easier to work with sums. Since the logarithm is a monotonically increasing function, maximising the likelihood is the same as maximising the log-likelihood. This simple trick transforms the product into a sum:

This should look familiar if you've worked with deep learning models. In training a model, we often aim to minimise a loss function. A very common loss function for classification problems is the Negative Log-Likelihood (NLL). Minimising the NLL is mathematically equivalent to maximising the log-likelihood. So when you're training a neural network with this loss, you are performing Maximum Likelihood Estimation to find the best weights for your model.

We know we want an unbiased estimator with the lowest possible variance. But is there a theoretical limit to how low that variance can be? Yes, there is. The Cramér-Rao Lower Bound (CRLB) provides a floor on the variance of any unbiased estimator. It tells us the best precision we can ever hope to achieve for a given estimation problem.

The quantity $I(\theta)$ is the Fisher Information, which measures how much information a random variable $X$ carries about the unknown parameter $\theta$. A higher Fisher Information means the data provides more clues about the parameter, allowing for a lower variance (more precise) estimate. An estimator that achieves this lower bound is called an efficient estimator.

These concepts form the bedrock of frequentist inference. Understanding the properties of estimators, the logic of maximum likelihood, and the theoretical limits of precision allows us to build, evaluate, and interpret machine learning models with mathematical rigour. When you choose a loss function or evaluate a model's performance, you are engaging with these foundational ideas.