Let's say a friend flips a coin and keeps the result hidden. Before they reveal it, how much do you know? Not much. The outcome could be heads or tails. Now, imagine they use a double-headed coin. Before they even flip it, you know the result will be heads. There's no uncertainty at all.

Information theory is the science of quantifying this uncertainty. It gives us a mathematical way to measure the 'surprise' in an event. A predictable event, like the double-headed coin, contains very little information. An unpredictable one, like a fair coin toss, contains more.

This idea, developed by Claude Shannon in the 1940s, forms the bedrock of everything from how we compress files to how we send data across the internet.

Shannon named his measure of uncertainty entropy. The more possibilities an event has, and the more evenly balanced their probabilities are, the higher the entropy. Higher entropy means more uncertainty and, therefore, more information is gained when we learn the outcome.

We can calculate it with a specific formula.

Let's apply this to our coins. A fair coin has two outcomes (heads, tails), each with a probability of 0.5. Its entropy is:

$H_{fair} = - (0.5 \log_2(0.5) + 0.5 \log_2(0.5)) = 1$ bit.

Now, consider a biased coin that lands on heads 90% of the time ($p=0.9$) and tails 10% of the time ($p=0.1$). Its entropy is:

$H_{biased} = - (0.9 \log_2(0.9) + 0.1 \log_2(0.1)) \approx 0.47$ bits.

The fair coin has higher entropy because its outcome is more uncertain. Learning the result of the fair coin toss gives you 1 bit of information, while learning the result of the biased toss gives you less than half a bit.

Events with just two possible outcomes are extremely common. Think yes/no, on/off, or heads/tails. This special case is described by the binary entropy function.

This function shows a beautiful and intuitive relationship. When the probability $p$ is 0 or 1, the outcome is certain, and the entropy is 0. There's no information because there's no surprise. The entropy is maximized when $p=0.5$, exactly the case of our fair coin flip.

What if we have two different variables? We might want to know how much information they share. For instance, how much does knowing the temperature today tell us about the temperature tomorrow? This is where mutual information comes in.

Mutual information, denoted $I(X;Y)$, measures the reduction in uncertainty about one variable, $X$, from observing another variable, $Y$. If two variables are completely independent (like the results of two separate coin flips), their mutual information is zero. Knowing one tells you nothing about the other. If they are perfectly correlated (knowing one means you know the other), the mutual information is equal to the entropy of either one.

Mathematically, it's defined using entropy.

In simple terms, mutual information is what you start with ($H(X)$) minus what you're left with ($H(X|Y)$). The result is the amount of uncertainty that was eliminated—the information that $Y$ provided about $X$.

Entropy has a very practical consequence: it sets the ultimate limit for data compression. Imagine you have a long text file. You can compress it into a ZIP file, making it smaller. How small can it get?

Shannon proved that the minimum average number of bits needed to represent a piece of information is equal to its entropy. This is known as lossless compression—no information is lost, and you can always perfectly reconstruct the original file.

Think about a file that's just the letter 'a' repeated a million times. The probability of seeing an 'a' is 1, and for any other character it's 0. The entropy is 0. This file is perfectly predictable and can be compressed dramatically (e.g., "one million 'a's").

Now consider a file full of truly random, independent characters. Each character is equally likely. This file has very high entropy. It's highly unpredictable and cannot be compressed much, if at all. There are no patterns for a compression algorithm to exploit.

This is why you can't just keep zipping a file to make it smaller and smaller. You eventually hit the fundamental limit set by its entropy.

Ready to test your knowledge?

These core ideas—entropy, mutual information, and the limits of compression—are the foundation of how we handle information in a digital world.