You already know that sample means tend to form a predictable normal distribution, thanks to the Central Limit Theorem. This gives us a map of what's likely and what's rare. Now, we'll use that map to test formal claims.

Statistical testing is like a structured argument with data. Think of it as a courtroom. There's a default assumption—innocent until proven guilty. In statistics, our default is that nothing has changed, or there's no effect. This is called the Null Hypothesis (H₀). It's the status quo, the boring reality we assume is true unless we find overwhelming evidence to the contrary.

The claim we want to test is the Alternative Hypothesis (Hₐ or H₁). This is our research question, our hunch, the thing we suspect might be true. The entire process is about collecting enough evidence from our sample to see if we can confidently reject the null hypothesis in favor of the alternative.

To test these ideas, we need to translate them from plain English into the precise language of mathematics. This involves stating our hypotheses in terms of population parameters like the mean ($μ$) or proportion ($p$).

Let's say a company's website has a historical click-through rate of 3% ($p=0.03$). They redesign a button and want to know if the new design is better. Their research question becomes a pair of formal hypotheses:

• Null Hypothesis (H₀): The new button has no effect on the click-through rate. The new rate is the same as the old one.
• Alternative Hypothesis (Hₐ): The new button improves the click-through rate. The new rate is higher.

Or, consider a factory that produces bolts with an average length of 50mm ($μ=50$). They recalibrate a machine and need to know if the average length has changed at all.

• H₀: The machine recalibration had no effect. The average bolt length is still 50mm.
• Hₐ: The machine recalibration changed the average length. It is no longer 50mm.

Notice the difference in the alternative hypotheses above. The first one used a “greater than” sign ($>$), while the second used a “not equal to” sign ($≠$). This distinction is crucial and defines the type of test we'll perform.

A one-tailed test is directional. It looks for an effect in one specific direction. Words like “increase,” “decrease,” “improve,” or “worse” in a research question signal a one-tailed test. We are only interested in whether the new value is specifically greater than or specifically less than the old value.

• $H_a: \mu > 50$ (right-tailed test)
• $H_a: \mu < 50$ (left-tailed test)

A two-tailed test is non-directional. It looks for any change at all. We use it when we care about a difference but don't have a prior assumption about its direction. A research question asking if something has “changed” or is “different from” the baseline calls for a two-tailed test.

• $H_a: \mu \neq 50$ (two-tailed test)

Choosing the right test is critical. A one-tailed test has more power to detect an effect in the specified direction, but it completely misses an effect in the opposite direction. A two-tailed test is more conservative but can detect a change no matter which way it goes. The choice should always be made before you look at the data, based purely on the research question.

Hypothesis testing uses a powerful logical tool: proof by contradiction . Instead of proving our alternative hypothesis directly, we start by assuming the null hypothesis is true. Then we look at our data.

We ask: “If the null hypothesis were true, how likely is it that we would get a sample result as extreme as the one we actually got?”

If our sample result is extremely unlikely under the null assumption (for example, if a coin lands on heads 18 times out of 20 flips), we have two possible conclusions:

1. The null hypothesis is true, and we just witnessed a very rare, fluke event.
2. The null hypothesis is false. Our initial assumption was wrong.

In statistics, we go with the second option. We reject the null hypothesis because the evidence makes it look absurd. This indirect method of disproving one thing to lend support to another is the logical engine of all statistical testing.

Now that you know how to formulate hypotheses, the next step is to define exactly how rare an event needs to be for us to reject the null hypothesis. This brings us to the concepts of p-values and significance levels.