A/B test statistics

Welcome! I'm excited to help you master the statistical machinery that powers modern product and marketing decisions.

Before we construct our custom learning path and dive into the math, I'd love to learn a little bit more about your background so I can tailor our pace and examples perfectly.

What brings you to learning A/B test statistics today (e.g., preparing for an interview, analyzing tests at work, or general interest), and how familiar are you already with basic statistics?

I'm preparing for an interview and have a basic stats background.

It's great to have you here! Since you already have a basic background in statistics and are preparing for an interview, we can move efficiently through the foundational theory and focus on how these concepts translate into real-world product decisions and common interview questions.

Let's take a look at our game plan for this session to make sure we cover all the core statistical machinery behind A/B testing.

Your path

P-Values and Confidence Intervals

Sample Size and Statistical Power

Practical Analysis and Common Pitfalls

Starting with the Foundations

Before we calculate anything, we have to set up the rules of the game. In any A/B test, we are trying to determine if a new change (like a redesigned checkout flow or a new algorithm) actually drives a meaningful change in user behavior, or if the difference we see is just random background noise.

To do this formally, we establish two competing statements: the and the alternative hypothesis.

The first key skill in product interviews is being able to frame these two hypotheses cleanly. Let's practice how to translate a product change into these formal mathematical assumptions.