Three fertilizers, three teaching methods, three carriers, the moment you have more than two groups, the two-sample t-test runs out. Testing every pair separately quietly multiplies your false-alarm rate. ANOVA (analysis of variance) asks the global question once, at a single α.
One omnibus test (is any mean different?) protects the error rate; one post-hoc test (which pairs?) gives the detail. F = between-group spread ÷ within-group spread.
Why Not Just Run Many t-Tests?
With k groups there are k(k−1)/2 possible pairs, 3 for three groups, 10 for five. Each t-test carries its own 5% false-alarm risk, and those risks compound. Run enough comparisons and you are almost guaranteed to "find" something that is not there.
The notebook makes it concrete: five identical groups still trigger at least one "significant" pairwise t-test about 40% of the time. ANOVA restores a single honest α by testing all means together before any pairwise question is asked.
The F Statistic: Between vs Within
ANOVA splits the total variation into two pieces: how far the group means sit from the grand mean (between-group), and how much the data scatters inside each group (within-group, the noise). The F statistic is their ratio.
Under H₀ (all means equal) the between- and within-group variances estimate the same thing, so F hovers near 1. When the means are genuinely separated relative to the noise, F shoots up. The p-value comes from the F distribution with (k − 1, N − k) degrees of freedom. In the notebook, three separated groups give F ≈ 25 (p ≈ 10⁻⁹), while three identical groups give F ≈ 1.
Post-Hoc: Which Pairs Differ?
A significant F is an omnibus verdict: some mean differs, but not which. The fix is a post-hoc test that compares every pair while holding the family-wise error at 5%. Tukey's HSD (Honest Significant Difference) is the standard choice.
Each Tukey interval is widened just enough that all comparisons together stay within the 5% budget. A pair is significant exactly when its confidence interval excludes 0. This is the disciplined sequel to a significant ANOVA, never a scatter of uncorrected t-tests.
Real-World Example: Delivery by Carrier
A shipper benchmarks three carriers on door-to-door delivery time, 80 shipments each. Do the carriers genuinely differ, and if so, which? One-way ANOVA answers the first question; Tukey HSD answers the second.
One row per shipment with carrier (three groups),
delivery_hours, and the destination region.
| Carrier | Mean delivery (h) | Tukey comparison | Different? |
|---|---|---|---|
| FastFreight | 30.4 | FastFreight vs GroundLink | yes (faster) |
| GroundLink | 35.9 | FastFreight vs RegionalCo | yes (faster) |
| RegionalCo | 37.4 | GroundLink vs RegionalCo | no clear difference |
The omnibus ANOVA gives F ≈ 22.9, p ≈ 10⁻⁹, so we reject the hypothesis that all three carriers are equally fast; carrier explains about 16% of the variation (η² ≈ 0.16). Tukey HSD then localizes it: FastFreight (≈ 30.4 h) is significantly faster than both GroundLink and RegionalCo, while GroundLink and RegionalCo are statistically indistinguishable. The decision is clear, route through FastFreight, and ANOVA-plus-Tukey is what justifies it rigorously.
First, though, the checks that make those p-values trustworthy. ANOVA assumes independent observations (true here, each shipment is separate), roughly equal variances across the groups, and roughly normal residuals. The companion notebook runs Levene's test (p ≈ 0.10, so the variances are close enough) and a residual QQ plot backed by Shapiro (p ≈ 0.28, near-normal), so the classic F-test is valid on this data. When a check fails you switch tools rather than drop the question: unequal variances call for Welch's ANOVA, and heavy non-normality for the rank-based Kruskal-Wallis test (which agrees here, p ≈ 3×10⁻⁹). And report the bias-corrected effect size ω² ≈ 0.15 next to the p-value: about 15% of delivery-time variance is down to the carrier.
ANOVA in Machine Learning & AI
Comparing more than two options, models, configs, arms, is exactly an ANOVA problem.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| One-way ANOVA | Comparing 3+ models or configs | accuracy across four architectures |
| Family-wise error | Why A/B/n tests need correction | 5 variants → many pairwise risks |
| Tukey HSD | Which model is the real winner | post-hoc pairwise model comparison |
| Between vs within variance | Signal vs noise in metrics | config effect vs run-to-run jitter |
Benchmarking several models or hyperparameter settings is a multiple-comparison minefield: with random seeds adding run-to-run noise, picking "the best" from many pairwise comparisons invites a lucky winner. A one-way ANOVA across configurations first asks whether any setting truly matters; if so, a post-hoc test identifies the genuine standouts while controlling the family-wise error. The same structure underlies A/B/n testing (more than two variants), where treating each arm-vs-control comparison independently inflates false positives unless you correct for it.
Run ANOVA and Tukey HSD in Python
The companion notebook demonstrates the multiple-comparisons trap, checks the ANOVA assumptions
(Levene for equal variance, a residual QQ plot and Shapiro for normality, plus a Kruskal-Wallis cross-check),
builds the F test with f_oneway, reports the full table with mean squares and
ω², runs Tukey HSD in one call with pairwise_tukeyhsd, and loads
anova--carrier_delivery.xlsx to compare three carriers and find which differ.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓ANOVA tests "all group means equal" in one shot, avoiding the error inflation of many pairwise t-tests.
- ✓The F statistic is between-group variance ÷ within-group variance; near 1 under H₀, large when groups separate.
- ✓A significant F is omnibus, follow it with Tukey HSD to see which specific pairs differ.
- ✓Real data: carriers differ (F ≈ 22.9, p ≈ 10⁻⁹, η² ≈ 0.16); FastFreight is significantly faster.
- ✓In ML/AI: comparing 3+ models or A/B/n arms is an ANOVA problem; correct for multiplicity.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
One-way ANOVA
Three groups N(20, 5, n = 30), N(22, 5, n = 30), N(26, 5, n = 30). Run a one-way ANOVA.
stats.f_oneway(g1, g2, g3).Effect size η²
For those groups compute η² = SSbetween / SStotal.
ANOVA = t-test for two groups
Show that for exactly two groups, F = t² (pooled).
f_oneway to ttest_ind(equal_var=True).Tukey HSD by hand
For the three groups in #1, find which pairs differ (α = 0.05) using the studentized range.
Real data: carriers
Load anova--carrier_delivery.xlsx and run the one-way ANOVA across carriers.
delivery_hours by carrier.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on ANOVA. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
So far the outcomes have been numeric. Chi-Square Tests turns to categorical data, testing whether counts match expectations and whether two categorical variables are related.