Suppose a coin lands heads 62 times in 100 tosses. Is it biased, or is 62 just a lucky run of a fair coin? A hypothesis test is the disciplined way to answer "could chance alone produce this?", and it runs the same four steps no matter what you are testing.
State H₀ and H₁, choose a test statistic, find its distribution assuming H₀ is true, then compute a p-value and decide. Every test in this Part is a variation on those four steps.
The Question Behind Every Test
A test is a courtroom for data. The null hypothesis is the presumption of innocence: nothing unusual is happening (the coin is fair, the drug does nothing, the two groups are equal). The alternative is the prosecutor's claim. We do not "prove" the alternative; we ask whether the evidence against the null is strong enough to abandon it.
The alternative can be two-sided (the value differs, in either direction) or one-sided (it is specifically higher, or specifically lower). "Is the coin biased?" is two-sided; "are the bottles under-filled?" is one-sided. Decide which before seeing the data, picking a side after the fact quietly doubles your false-alarm rate.
The Test Statistic & the Null World
A test statistic compresses the data into one number measuring distance from the null, usually in standard-error units (a z or a t). To judge whether that number is large, we need its sampling distribution under H₀: what values it would take if the null were true. For the coin, we can literally simulate thousands of fair-coin experiments.
In the notebook the fair-coin simulation centers neatly on 0.50, and our observed 0.62 lands about 2.4 standard errors into the tail. The standardized statistic, z = (p̂ − 0.5) / SE, turns "62 heads" into a universal yardstick we can read off any normal table. The further out the statistic, the harder the null is to believe.
The p-value & the Decision Rule
The p-value is the probability, computed under H₀, of a result at least as extreme as the one observed. Small p means "the data would rarely happen by chance", evidence against the null. We fix a significance level α (often 0.05) in advance and reject H₀ when p < α.
For the coin the two-sided p-value is about 0.016, below 0.05, so we reject the fair-coin hypothesis. The crucial subtlety: a p-value is not the probability that H₀ is true. It is the probability of data this extreme assuming H₀ is true, a statement about the data, not the hypothesis.
| Reality ↓ / Decision → | We reject H₀ | We fail to reject H₀ |
|---|---|---|
| H₀ is true | Type I error (false alarm, rate α) | correct |
| H₀ is false | correct (a real detection) | Type II error (a miss, rate β) |
"Fail to reject" is not the same as "accept", absence of evidence is not evidence of absence. The next chapter is devoted to these two error types, α, and the test's power to catch real effects.
Real-World Example: A Short-Fill Investigation
A bottling line claims every bottle holds 500 ml. Quality control samples 45 bottles; the average looks a touch low. Is the line genuinely under-filling, or is this within normal variation? This is the first real test built from the four-step logic, a one-sample t-test.
One row per sampled bottle with fill_ml, plus the line and
shift it came from.
| Step | Result |
|---|---|
| Hypotheses | H₀: μ = 500 ml vs H₁: μ < 500 ml (one-sided) |
| Sample | n = 45, mean ≈ 498.1 ml, sd ≈ 5.05 |
| Test statistic | t = (498.1 − 500) / SE = −2.57 (df = 44) |
| p-value | one-sided ≈ 0.007 (two-sided ≈ 0.014) |
| 95% CI for μ | [496.6, 499.6] ml — lies below 500 |
| Decision | p < 0.05 → reject H₀: the shortfall is real |
The one-sided p-value of about 0.007 is well under 0.05, so we reject the 500 ml claim: this line really is under-filling. Notice the confidence interval [496.6, 499.6] tells the same story from the estimation side, it sits entirely below 500. The effect is modest (Cohen's d ≈ −0.38), which is exactly why a formal test, not a glance at the average, was needed to call it.
Hypothesis Testing in Machine Learning & AI
The reject/fail-to-reject logic underpins how modern ML systems are evaluated, shipped, and trusted.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| H₀ vs H₁ | "Is model B actually better than model A?" | online A/B test of two models |
| Null distribution | Permutation / bootstrap null | shuffle labels to get a no-effect baseline |
| p-value & α | Ship/no-ship decision gate | launch only if lift clears significance |
| One- vs two-sided | Guardrail vs improvement metrics | "not worse" vs "strictly better" |
Every time a team claims a new model "beats" the baseline, there is an implicit hypothesis test: the improvement on a benchmark could be a real gain or just sampling noise from a particular test set. Treating the difference as a test of H₀: no improvement, with a p-value or a confidence interval on the gap, is what separates a genuine advance from a lucky split. The same logic governs A/B testing of recommendation and ranking systems, where the decision gate ("ship only if p < α") protects against rolling out changes that merely looked good by chance.
Run the four-step logic in Python
The companion notebook sets up H₀ and H₁ for a coin, simulates the null world, computes a
p-value two ways (simulation and formula), and then loads the-logic-of-hypothesis-testing--factory_fills.xlsx to run the
one-sample t-test of the 500 ml claim end to end.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Four steps: state H₀/H₁, pick a test statistic, find its null distribution, compute a p-value and decide.
- ✓H₀ is the default of no effect; we reject it only with strong evidence, like a presumption of innocence.
- ✓The p-value is the chance of data this extreme if H₀ is true, not the probability that H₀ is true.
- ✓Real data: fill volumes average 498.1 ml, t = −2.57, one-sided p ≈ 0.007, so the line really under-fills.
- ✓In ML/AI: "is B better than A?" is a hypothesis test; the p-value is the ship/no-ship gate.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and SciPy before checking the solutions.
State the hypotheses
A coach claims a drill raises free-throw accuracy above 70%. Write H₀ and H₁ and say whether it is one- or two-sided.
Build a null distribution
Simulate 50 fair coins 20,000 times and find the 2.5th and 97.5th percentiles of the heads-fraction.
rng.binomial(50, 0.5, 20000)/50.p-value by simulation
You see 33 heads in 50 tosses. Compute the two-sided p-value under a fair coin by simulation.
One-sample t from scratch
For n = 40, mean 51.2, sd 9, test H₀: μ = 50 (two-sided). Compute t and p, then verify with SciPy.
Real data: the short-fill test
Load the-logic-of-hypothesis-testing--factory_fills.xlsx and run the one-sided one-sample t-test of H₀: μ = 500.
stats.ttest_1samp, then halve the p-value for a one-sided H₁.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on the logic of hypothesis testing. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You now have the four-step skeleton. Significance, p-values & Errors examines what that p-value and the threshold α actually promise, the two ways a test can be wrong (Type I and Type II), and the power to detect real effects.