In everyday language, "information" is about facts and meaning. But in information theory, it has a much more precise definition. Information is a measure of surprise. It's the resolution of uncertainty.

Think about a coin flip. Before the coin lands, you don't know the outcome. There's uncertainty. When it lands on heads, that uncertainty is gone. You've received one "bit" of information. The less likely an event is, the more surprising it is, and the more information you gain when it happens.

To measure surprise, we need to know how likely things were to happen in the first place. This is where probability comes in. An event with a low probability is surprising, so it carries a lot of information. An event with a high probability is expected, so it carries very little information.

Consider two coins. One is a fair coin, with a 50% chance of heads and 50% of tails. The other is a biased coin that lands on heads 99% of the time. Which coin flip gives you more information on average? The fair one. With the biased coin, you're almost certain it's going to be heads. Learning that it is, in fact, heads isn't much of a surprise.

We can formalize this relationship with a simple formula. The information, $I$, of an event with probability $p$ is calculated like this:

Let's apply this to our fair coin, where the probability of heads is 0.5. The information gained from seeing a heads is:

$I(\text{heads}) = -\log_2(0.5) = -(-1) = 1$ bit.

The same is true for tails. Each equally likely outcome gives you exactly one bit of information.

While information measures the surprise of a single event, we often want to measure the total uncertainty of a system with many possible outcomes. This average uncertainty is called entropy.

Entropy is the expected amount of information from an event. You calculate it by taking the information of each possible outcome, multiplying it by the probability of that outcome, and adding them all up.

The formula for entropy, usually represented by $H$, looks like this for a set of outcomes $X$:

For our fair coin (50/50), the entropy is:

$H = - (0.5 \times \log_2(0.5) + 0.5 \times \log_2(0.5)) = 1$ bit.

For a biased coin that's 99% heads, the entropy is much lower, about 0.08 bits. This confirms our intuition: the biased coin's outcome is far more predictable, so the system has less uncertainty, and therefore, lower entropy. Maximum entropy occurs when all outcomes are equally likely, because that's the point of maximum uncertainty.

This concept is the bedrock of information theory. By quantifying uncertainty with entropy, we can begin to understand the fundamental limits of how much data can be compressed and how reliably it can be sent over a noisy channel.

Understanding information, probability, and entropy gives us a powerful framework for analyzing communication and data in any form.