You're likely familiar with the ritual of Null Hypothesis Significance Testing (NHST). You set up a null hypothesis ($H_0$), which assumes no effect, and an alternative hypothesis ($H_1$), which claims there is one. You collect data, run a test, and get a p-value. If that p-value is small (typically < 0.05), you reject the null and declare your finding "statistically significant."

But a significant result doesn't automatically mean an important one. This framework is just the starting point. Relying solely on a p-value is like judging a movie by its poster. It gives you a hint, but it misses the entire plot. To truly understand your results, we need to ask more nuanced questions: How certain are we? How big is the effect? And what are the risks of being wrong?

Every hypothesis test is a judgment call based on incomplete data. Because we're working with samples, not entire populations, there's always a chance we'll make the wrong decision. There are two specific ways this can happen.

A Type I error is a false positive. It happens when you reject the null hypothesis when it was actually true. Think of it as a smoke alarm going off when there's no fire. The probability of this error is denoted by alpha (α), which is the significance level you set for your test (e.g., 0.05).

A Type II error is a false negative. This occurs when you fail to reject the null hypothesis when it was actually false. This is the fire that the smoke alarm misses. The probability of a Type II error is denoted by beta (β). These two errors have an inverse relationship; reducing the chance of one often increases the chance of the other.

This brings us to statistical power, which is arguably one of the most important concepts in experimental design. Power is the probability that you will correctly detect an effect that actually exists. In other words, it's the probability of avoiding a Type II error.

Mathematically, it's simple:

A study with high power has a good chance of leading to a correct conclusion when the alternative hypothesis is true. Researchers aim for high power, often 80% or more, by increasing sample size, using more precise measurements, or choosing a stronger experimental design.

Statistical significance can be misleading. With a large enough sample size, even a minuscule, trivial effect can produce a tiny p-value. For example, a new diet pill might be proven to cause statistically significant weight loss, but if that loss is only half a pound over six months, is it practically meaningful?

This is where effect size comes in. It's a standardized measure of the magnitude of an effect, independent of sample size. It tells you not just if there's a difference, but how big that difference is. One of the most common measures is Cohen's d.

By convention, a Cohen's d of 0.2 is considered a 'small' effect, 0.5 a 'medium' effect, and 0.8 a 'large' effect. Reporting effect sizes alongside p-values gives a much fuller picture of your findings, separating statistical noise from substantive results.

Imagine you're testing 20 different hypotheses at once, each with a significance level of α = 0.05. Remember, this alpha level means you have a 5% chance of a Type I error (a false positive) for any single test. When you run multiple tests, the odds of getting at least one false positive add up quickly.

This is known as the multiple comparisons problem. If you test enough things, you're almost guaranteed to find a 'significant' result just by random chance. This is one way that p-hacking occurs, where researchers run test after test until they find a p-value under 0.05, even if the finding is spurious.

To combat this, statisticians use corrections that adjust the p-value threshold. The simplest and most famous of these is the Bonferroni correction.

So, if you run 10 tests, your new threshold for significance becomes 0.05 / 10 = 0.005. Only p-values below this much stricter level would be considered significant. While effective at reducing Type I errors, the Bonferroni correction is very conservative and can increase the risk of Type II errors—missing a real effect. More advanced methods exist, but it serves as a crucial reminder to be skeptical of single 'significant' findings when many tests have been run.

Moving beyond p-values to consider power, effect sizes, and the risks of multiple comparisons allows for a more robust and honest interpretation of data. It shifts the goal from simply finding a 'significant' result to understanding the real-world importance and reliability of your work.