In everyday life, “information” is a fuzzy concept. It can mean facts, news, or knowledge. But in the 1940s, a mathematician named Claude Shannon gave it a precise, mathematical meaning that kickstarted the digital revolution.

In information theory, information isn’t about meaning or importance. It’s a measure of surprise. The more surprising an event is, the more information it carries. Learning that the sun rose this morning is not very surprising, so it contains very little information. Learning that it will snow in the Sahara desert tomorrow would be a massive surprise, carrying a huge amount of information.

This simple idea—that information is the reduction of uncertainty—is the bedrock of how we store, transmit, and process data today.

To work with this idea, we need a way to quantify it. This is where the concept of self-information comes in. It’s a score we assign to an event based on how unlikely it is to happen. The lower the probability, the higher the self-information.

The formula looks like this:

Let's break that down:

• $I(x)$ is the self-information of an event $x$.
• $P(x)$ is the probability that event $x$ occurs.
• $\log_2$ is the base-2 logarithm. We use it for a special reason: it measures information in units called bits.
• The negative sign ensures the result is a positive number, since the logarithm of a probability (a value between 0 and 1) is always negative.

One bit is the amount of information you get from a single, fair coin flip. The outcome is completely uncertain. There's a 50% chance of heads and a 50% chance of tails. When you see the result, your uncertainty is cut in half. Let's plug it into the formula:

The relationship is clear: as probability goes down, information goes up. Imagine a coin that is heavily biased to land on heads.

Let's say the probability distribution for this biased coin is:

• $P(\text{heads}) = 0.9$
• $P(\text{tails}) = 0.1$

Here, you already expect it to land on heads. Seeing that happen doesn't resolve much uncertainty.

That's very little information. But seeing the rare outcome—tails—is a big surprise.

Seeing the unlikely event gives you over 20 times more information than seeing the likely one.

This principle applies everywhere. A common word like "the" in an English sentence carries less information than a rare word like "archaeopteryx." A sunny day in a desert provides less information than a rainy one. Information theory gives us the tools to put a number on just how much less.

This fundamental connection between probability and information is the key to understanding how data works at a deeper level.