Introduction to Bayesian Statistics
Introduction to Probability
The Measure of Certainty
Probability is simply a way to measure how likely something is to happen. We use it every day, even if we don't put numbers to it. When you check the weather forecast and see a 90% chance of rain, you're using probability to decide whether to bring an umbrella.
In mathematics, we pin this down more formally. The probability of an event is a number between 0 and 1. A probability of 0 means the event is impossible. A probability of 1 means the event is absolutely certain to happen. Everything else falls somewhere in between. A coin flip has a 0.5 probability of landing on heads, meaning it's equally likely to be heads or tails.
The Rules of the Game
To make sure our calculations are consistent, probability follows three basic rules, often called axioms. They’re quite intuitive.
Rule 1: Probabilities are never negative. The lowest an event's probability can be is 0 (impossible). It can't be -0.5.
Here, means "the probability of event A happening."
Rule 2: The probability of all possible outcomes combined is 1. If you roll a standard six-sided die, you are certain to roll a 1, 2, 3, 4, 5, or 6. The sum of the probabilities for all possible outcomes must equal 1.
Here, represents the sample space, which is the set of all possible outcomes.
Rule 3: If two events can't happen at the same time, their combined probability is the sum of their individual probabilities. You can't roll a 2 and a 4 on a single die roll. These are called mutually exclusive events. The probability of rolling a 2 or a 4 is simply the probability of rolling a 2 plus the probability of rolling a 4.
These three rules are the foundation upon which all of probability theory is built.
Changing the Odds
Sometimes, knowing that one event has occurred changes the probability of another. This is called conditional probability. Imagine you have a standard deck of 52 cards. The probability of drawing a King is , or about 7.7%.
But what if you know that the card you drew is a face card (a Jack, Queen, or King)? Now you're not choosing from 52 cards anymore. Your sample space has shrunk to just the 12 face cards. Within this smaller group, there are 4 Kings. So, the probability of drawing a King given that it's a face card is now , or 33.3%.
We write this with a vertical bar, , which reads "the probability of A given B." The formula looks like this:
In our example:
- Event A is drawing a King.
- Event B is drawing a face card.
is the probability of drawing a card that is both a King and a face card. Since all Kings are face cards, this is just the probability of drawing a King, which is .
is the probability of drawing a face card, which is .
So, . New information changed the odds.
Variables with a Chance
Often, we're not interested in the outcomes themselves, but in a numerical value associated with them. For example, in a game where you get $10 for rolling a 6 on a die and $0 for any other number, you care about the money, not just the number on the die.
This is where a random variable comes in. A random variable is a variable whose value is a numerical outcome of a random phenomenon. We usually represent it with a capital letter, like .
Random Variable
noun
A variable whose value is a numerical outcome of a random process.
For our die game, we can define a random variable :
- if the die roll is 6
- if the die roll is 1, 2, 3, 4, or 5
The random variable connects the outcomes of our experiment (the die roll) to numerical values we care about (the money). We can then analyze the probabilities associated with these values. For instance, the probability that is , and the probability that is . Random variables are a crucial tool for modeling and understanding the numerical results of uncertain situations.
Now that you understand the basics, let's test your knowledge.
If an event is absolutely certain to happen, what is its probability?
A bag contains 3 red marbles and 7 blue marbles. You randomly draw one marble and see that it's blue. You do not replace it. What is the probability that the next marble you draw is also blue?