Intro to Hypothesis Testing Formulate a statement to 'prove something wrong' by testing a default assumption (Null Hypothesis) against a new claim (Alternative Hypothesis), much like a court trial.

P-value & Significance The p-value measures how likely your results are due to random chance; if it's below a set threshold (alpha), you reject the default assumption as unlikely.

Comparing Averages (Z & T-tests) Z-tests and T-tests are used to compare the averages of one or two groups, with the choice depending on sample size and whether the population variance is known.

Comparing >2 Groups (ANOVA) ANOVA compares the means of three or more groups at once by analyzing their variances, which avoids the increased error risk of running multiple separate tests.

Categorical Data (Chi-Squared) The Chi-Squared test checks if a relationship exists between two categorical variables by comparing the frequencies you observe to the frequencies you would expect.

Non-parametric Tests When your data is skewed or doesn't follow a normal distribution, non-parametric tests use ranks instead of actual values to provide robust group comparisons.

Intro to Hypothesis Testing

1. Null Hypothesis (H₀): This is the default assumption or status quo. It always contains a statement of equality (=, ≤, or ≥).
2. Alternative Hypothesis (H₁ or Hₐ): This is the new claim you want to test. It contradicts the null and contains a statement of inequality (≠, <, or >).
3. Innocent Until Proven Guilty: We assume the null hypothesis is true unless the data provides overwhelming evidence against it.
4. We Never "Prove" the Alternative: We only find evidence to reject the null hypothesis or fail to reject it. We don't prove things true.
5. Proof by Contradiction: The core logic is to show that the observed data is extremely unlikely if the null hypothesis were true, so we reject it.

P-value & Significance

1. P-value Definition: The probability of observing your data (or something more extreme) if the null hypothesis is true. It is a measure of surprise.
2. Small P-value: A small p-value (e.g., < 0.05) means your results are surprising and unlikely to be due to random chance alone, leading you to reject H₀.
3. Large P-value: A large p-value (e.g., > 0.05) means your results are not surprising and are consistent with random chance, so you fail to reject H₀.
4. Alpha (α) - Significance Level: This is a pre-determined threshold for "surprise" (commonly 0.05 or 5%). It's the risk you're willing to take of being wrong.
5. The Decision Rule: If P-value ≤ α, reject the null hypothesis. If P-value > α, fail to reject the null hypothesis.

Comparing Averages (Z & T-tests)

1. Purpose: Both tests compare the mean(s) of numerical data.
2. Z-test vs. T-test: Use a Z-test when you have a large sample size (typically n > 30) or know the population standard deviation. Use a T-test for small samples (n < 30) when the population standard deviation is unknown.
3. T-distribution: The t-distribution is wider than the normal (Z) distribution to account for the extra uncertainty of smaller samples.
4. One-Sample Test: Compares the mean of a single group to a known standard or hypothesized value (e.g., "Is the average bottle fill volume 500ml?").
5. Two-Sample Test: Compares the means of two different groups to see if they are different from each other (e.g., "Do users on Plan A spend more than users on Plan B?").

Comparing >2 Groups (ANOVA)

1. ANOVA's Purpose: Use Analysis of Variance (ANOVA) to compare the means of three or more groups simultaneously.
2. Avoid Multiple T-tests: Running many t-tests increases the Family-Wise Error Rate—the probability of making at least one false discovery (a Type I error).
3. How it Works: ANOVA analyzes the variance between groups compared to the variance within each group.
4. The F-statistic: This is the ratio of between-group variability to within-group variability. A large F-statistic suggests the group means are different.
5. One-Way ANOVA: This is the simplest form, used when you have one categorical independent variable and one quantitative dependent variable (e.g., comparing test scores across three different teaching methods).

Categorical Data (Chi-Squared)

1. Use Case: Use the Chi-Squared (χ²) test for categorical data, not numerical averages.
2. Test for Independence: The most common use is to determine if two categorical variables are independent or related (e.g., "Is there a relationship between a person's region and their preferred product?").
3. Observed vs. Expected: The test works by calculating what frequencies you'd expect to see if there were no relationship, and comparing that to what you actually observed.
4. Large χ² value: A large Chi-Squared value indicates a significant difference between your observed and expected counts, leading you to reject the null hypothesis of independence.
5. Contingency Table: Data for a Chi-Squared test is typically organized in a contingency table, with rows for one variable and columns for the other.

Non-parametric Tests

1. When to Use: Use these tests when your data does not meet the assumptions of parametric tests (like Z-tests, T-tests, ANOVA), especially the assumption of normal distribution.
2. Rank-Based Logic: Instead of using the actual data values, these tests convert the values to ranks (1st, 2nd, 3rd, etc.) and perform calculations on those ranks.
3. Mann-Whitney U Test: The non-parametric equivalent of a two-sample T-test. It tests if two independent samples come from the same distribution.
4. Wilcoxon Signed-Rank Test: The non-parametric equivalent of a paired T-test. It's used for two related samples or repeated measurements on a single sample.
5. Robust to Outliers: Because they use ranks, non-parametric tests are less affected by extreme outliers or skewed data, making them safer for 'dirty' real-world data.

Z-test vs. T-test

Memory trick: Think T for Tiny samples. If your sample is small, you're more likely to use a T-test.

Topic: Comparing Averages (Z & T-tests)

Parametric vs. Non-parametric Tests

Memory trick: Parametric tests have more pre-requisites (assumptions). Non-parametric tests are your plan B when the data doesn't play by the rules.

Topic: Non-parametric Tests

❌ MISTAKE: Believing a large p-value proves the null hypothesis is true.

• Why it happens: It's easy to think of the test as a two-way switch. If a small p-value rejects the null, a large one must accept it.
• ✅ Instead: A large p-value simply means you don't have enough evidence to reject the null. It's the difference between a verdict of "not guilty" (not enough evidence to convict) and "innocent" (proven to have not done it). Statistics provides a "not guilty," not a proof of innocence for H₀.
• Topic: P-value & Significance

❌ MISTAKE: Running multiple T-tests to compare three or more groups.

• Why it happens: If a T-test compares two groups, it seems logical to just run more of them for more groups (A vs B, B vs C, A vs C).
• ✅ Instead: Use ANOVA. Each T-test has a chance of a false positive (e.g., 5%). Running multiple tests inflates this total error rate. If you run three tests, your chance of at least one false positive is not 5%—it's closer to 14%. ANOVA is designed to test all groups at once while keeping the error rate at your chosen alpha level (e.g., 5%).
• Topic: Comparing >2 Groups (ANOVA)

❌ MISTAKE: Choosing a test after looking at the results.

• Why it happens: It's tempting to "p-hack" by trying different tests until you find one that gives a significant result.
• ✅ Instead: You must define your null and alternative hypotheses, choose your significance level (α), and select your statistical test before analyzing the data. This is a core principle of the scientific method to prevent biased conclusions.
• Topic: Intro to Hypothesis Testing

❌ MISTAKE: Assuming statistical significance equals practical importance.

• Why it happens: The word "significant" sounds important. With very large datasets, even tiny, meaningless differences can become statistically significant.
• ✅ Instead: Always consider the effect size. A new drug might be statistically proven to reduce headaches by 0.1%—but is that a meaningful improvement for patients? Statistical significance tells you if an effect is likely real; it doesn't tell you if it's large enough to matter.
• Topic: P-value & Significance

The 5 Steps of Hypothesis Testing

Step 1: State the Hypotheses

• What happens: Define the Null Hypothesis (H₀, the status quo) and the Alternative Hypothesis (H₁, your claim).
• Key indicator: You will have two competing, mutually exclusive statements. One must contain an equals sign (H₀: μ = 100), the other an inequality (H₁: μ ≠ 100).

Step 2: Choose the Significance Level (α) and Test

• What happens: Decide on your risk threshold (α, usually 0.05). Based on your data type, sample size, and hypothesis, select the correct statistical test (e.g., T-test, ANOVA).
• Key indicator: You have a clear alpha value and a named test before you proceed.

Step 3: Calculate the Test Statistic and P-value

• What happens: Use your sample data to compute a test statistic (e.g., a t-score or F-statistic). This value is then used to find the corresponding p-value.
• Key indicator: You have a single number for the test statistic and another for the p-value.

Step 4: Make a Decision

• What happens: Compare your p-value to your alpha level.
• Key indicator: Follow the simple rule: If p-value ≤ α, you reject the Null Hypothesis. If p-value > α, you fail to reject the Null Hypothesis.

Step 5: State the Conclusion in Context

• What happens: Translate your statistical decision back into plain language related to the original problem.
• Key indicator: Your conclusion does not use statistical jargon. Instead of "We reject the null hypothesis," you say, "There is sufficient evidence to conclude that the new teaching method improves student scores."

Topic: Intro to Hypothesis Testing

Choosing the Right Statistical Test

Step 1: Identify your Data Type

• What happens: Determine if you are working with numerical (means, averages) or categorical (proportions, counts) data.
• Key indicator: Are you comparing average scores or counting how many people chose 'Yes' vs 'No'?

Step 2: Determine the Number of Groups

• What happens: Count how many groups or samples you are comparing.
• Key indicator: Is it one group against a standard? Two groups against each other? Or three or more groups?

Step 3: Check Assumptions (Especially Normality)

• What happens: Assess if your data follows a normal (bell-curve) distribution. You can do this with a visual plot (histogram) or a formal test.
• Key indicator: Is the data roughly symmetrical, or is it heavily skewed or full of outliers?

Step 4: Use a Decision Flow

• What happens: Follow a logical path:
  • Comparing 1-2 group means + Normal data → T-test or Z-test.
  • Comparing 3+ group means + Normal data → ANOVA.
  • Comparing categorical data (2+ categories) → Chi-Squared Test.
  • Comparing group means + Skewed/Non-normal data → Non-parametric test (e.g., Mann-Whitney U).

Topic: Non-parametric Tests