Not all information is created equal. A message telling you the sun will rise tomorrow morning contains very little new information. You already knew that. But a message correctly predicting the exact winning lottery numbers contains a massive amount of information. The difference lies in probability.

The lottery numbers are surprising because they are incredibly unlikely. The sunrise is not surprising because it's virtually certain. In the 1940s, a mathematician and engineer named Claude Shannon realized this connection was key to understanding information itself. He proposed that the amount of information in a message is directly related to its level of surprise, or how improbable it is.

Consider a simple coin flip. If the coin is fair, the probability of heads is 0.5 and tails is 0.5. There's a genuine uncertainty about the outcome. Now, imagine a biased coin that lands on heads 99% of the time. If someone tells you it landed on heads, you're not very surprised. But if it lands on tails, that's a significant piece of information. Shannon developed a way to measure this average uncertainty for any given system. He called it entropy.

Information entropy, often represented by $H$, gives us a precise number for the average amount of information produced by a random process. For a set of possible outcomes $X$, the entropy is calculated by taking the probability $p(x)$ of each outcome, multiplying it by its own logarithm, and summing the results.

The standard unit for information entropy is the "bit." A process with an entropy of 1 bit has the same average uncertainty as a fair coin flip.

Let's apply this to our coins. For a fair coin, the entropy calculation is:

The outcome of a fair coin flip contains exactly one bit of information. This is the maximum possible entropy for a two-outcome system. Maximum uncertainty occurs when all outcomes are equally likely.

For our biased coin (99% heads, 1% tails), the entropy is much lower:

The result is a tiny fraction of a bit. This makes intuitive sense. Because the outcome is almost always heads, there's very little uncertainty to resolve. The lower the entropy, the more predictable a system is.

This might seem like a purely academic exercise, but the concept of entropy is the bedrock of our digital world. Two major applications are data compression and communication.

Think about the English language. Letters don't appear with equal probability. 'E' is very common, while 'Z' is rare. Furthermore, some patterns are highly predictable. If you see a 'q', you're almost certain the next letter will be a 'u'. This predictability means English text has a relatively low entropy. A purely random string of letters would have a much higher entropy.

Data compression algorithms, like the ones used in ZIP files, work by finding these patterns and predictable structures. They create a shorthand to represent the data, effectively removing redundancy. The entropy of a piece of data tells us the absolute best-case limit for this compression. You can't compress a file to be smaller than its actual information content, as measured by its entropy.

Similarly, in communications, understanding the entropy of a signal helps engineers design systems that can transmit information efficiently and reliably over a noisy channel, like a cell phone network. It defines the fundamental rate at which information can be sent without error.

Let's test your understanding of these core concepts.

From coin flips to data compression, entropy provides a powerful mathematical lens for understanding uncertainty and information.