How do you change your mind when you learn something new? It seems simple, but there's a formal way to do it. Bayes' Theorem gives us a mathematical rule for updating our beliefs in light of new evidence. It's a cornerstone of probability theory that helps us move from an initial guess to a more informed position.

This idea is central to many machine learning models, especially those used for classification. At its heart, the theorem is a formula for reversing a conditional probability. If you know the probability of seeing certain evidence given a specific cause, Bayes' Theorem helps you find the probability of that cause being true, given that you've seen the evidence.

Formulas can be abstract, so let's use a common example: medical testing. Imagine a rare disease that affects 1 in every 10,000 people. There's a test for this disease that is 99% accurate.

Someone tests positive. What is the actual probability they have the disease? It might seem like it's 99%, but let's use Bayes' Theorem to see what's really going on.

First, let's define our terms:

• A: The person has the disease.
• B: The person tests positive.

Now we can assign probabilities based on what we know.

The one missing piece is $P(B)$, the total probability of anyone testing positive. A person can test positive in two ways: they have the disease and test positive (a true positive), or they don't have the disease and test positive (a false positive). We add these two possibilities together:

$P(B) = P(B|A)P(A) + P(B|A')P(A')$ $P(B) = (0.99 \times 0.0001) + (0.01 \times 0.9999) \approx 0.010098$

Now we have everything we need to plug into Bayes' Theorem to find $P(A|B)$, the probability of having the disease given a positive test.

The result is about 0.98%, or just under 1%. Even with a positive result from a 99% accurate test, the chance of actually having this rare disease is surprisingly low. This is because the disease is so rare that most positive tests are actually false positives.

Let's formalize the key terms we just used.

In our example, the prior was $P(A) = 0.0001$. This was our knowledge before the test result came in.

This was $P(B|A) = 0.99$. It measures how well the evidence (a positive test) supports our hypothesis (the person has the disease).

The posterior, $P(A|B) \approx 0.0098$, is our updated belief. If we got more evidence, like a second positive test, this posterior would become the prior for our next calculation. This is how Bayesian reasoning allows us to update our knowledge incrementally as we learn more.