What is information? In everyday life, we think of it as facts, news, or knowledge. But in information theory, the term has a very precise, mathematical meaning. It’s not about the content or meaning of a message, but about how much surprise it contains.

A message that tells you something you already knew contains zero information. A message that tells you something highly unexpected contains a lot of information. Think about the weather. If a forecast for a hot day in the Sahara desert says "sunny," you've learned very little. The outcome was almost certain. But if it says "snow," you've learned a great deal. The outcome was extremely unlikely.

Probability is the key to measuring this surprise. The lower the probability of an event, the more information its occurrence provides.

We can put a number on this. The amount of information, or "self-information," of a specific outcome $x$ is calculated based on its probability, $P(x)$. The formula uses a base-2 logarithm, and the result is measured in bits.

Let’s break this down. The logarithm helps because it makes information additive. If you have two independent events, the information you get from observing both is the sum of their individual information values. The negative sign is there simply to make the result positive, since the logarithm of a probability (a number between 0 and 1) is always negative.

Consider a fair coin flip. The probability of heads is $P(\text{heads}) = 0.5$. The information you get from seeing heads is:

$I(\text{heads}) = -\log_2(0.5) = -(-1) = 1$ bit.

This makes sense. A single bit, a 0 or a 1, is all you need to communicate the result of a coin flip.

While self-information measures the surprise of a single outcome, we often want to measure the uncertainty of an entire system or a random variable. This is where entropy comes in. Entropy is the average amount of information we get from one observation of a random variable. It's the expected value of the self-information.

The formula for entropy, denoted by $H$, looks at the probability of every possible outcome.

In this formula, we're summing up the information of each outcome ($-\log_2 P(x_i)$) weighted by its own probability ($P(x_i)$). A system has the highest entropy when all outcomes are equally likely, because that's when we are most uncertain about what will happen next. For our fair coin, with two outcomes each having a probability of 0.5, the entropy is:

$H = -(0.5 \times \log_2(0.5) + 0.5 \times \log_2(0.5)) = -(0.5 \times -1 + 0.5 \times -1) = 1$ bit.

This confirms our intuition. A fair coin flip, the classic example of a binary choice, contains exactly one bit of information. If the coin were biased, say, 75% heads and 25% tails, the entropy would be lower because we are less uncertain. We're pretty sure it's going to be heads.

As the graph shows, entropy is zero when the outcome is certain ($p=0$ or $p=1$) and reaches its maximum when the outcome is most unpredictable ($p=0.5$). This mathematical concept of entropy is the bedrock of information theory, allowing us to quantify the very nature of information itself.

Ready to check your understanding?

These core ideas—information as surprise and entropy as uncertainty—are the foundation for everything from how we compress files to how we send data across the internet.