The hardest part of testing is rarely the arithmetic, the computer does that. It is choosing which test fits the question and the data. Get that right and the rest is a function call; get it wrong and even a perfect calculation gives a meaningless answer.
Answer four questions, read the test off the map, check the assumptions. The framework chooses the test; the data decide the verdict, including the perfectly valid "no significant difference".
The Four Questions
Selecting a test is mechanical once you interrogate the problem. Ask, in order:
- Outcome type? Numeric (a measurement) or categorical (a label / yes-no)?
- How many groups? One vs a benchmark, two groups, or three or more?
- Independent or paired? Different subjects in each group, or the same subjects measured twice?
- Do assumptions hold? Roughly normal (or large n) and adequate expected counts, or skewed/ordinal (use the rank-based twin)?
Those four answers pin down the test almost uniquely. The whole point of Chapters 76-82 was to stock the toolbox; this chapter is the index that tells you which tool to grab.
The Decision Map
Here is the entire toolbox on one page. Find your row by outcome type, group count, and design; the rightmost column names the test (and its fallback, usually rank-based, when assumptions fail).
| Outcome | Groups | Design | Test (fallback) |
|---|---|---|---|
| Numeric | 1 vs a value | — | one-sample t-test |
| Numeric | 2 | independent | two-sample (Welch) t-test (Mann-Whitney) |
| Numeric | 2 | paired | paired t-test (Wilcoxon signed-rank) |
| Numeric | 3+ | independent | one-way ANOVA (Kruskal-Wallis) |
| Proportion | 1 | — | one-proportion z-test |
| Proportion | 2 | independent | two-proportion z-test (Fisher's exact) |
| Categorical | 2 variables | — | chi-square test of independence |
Assumptions & Robustness
The fourth question, assumptions, is what splits a row between its parametric test and its rank-based twin. A quick checklist before you commit:
| Check | If it holds | If it fails |
|---|---|---|
| Roughly normal (or n ≥ ~30)? | t-test / ANOVA | Mann-Whitney / Kruskal-Wallis |
| Free of heavy skew and outliers? | mean-based test OK | rank-based test (robust) |
| Expected counts ≥ ~5 per cell? | chi-square | exact test (e.g. Fisher's) |
| Data paired? | paired test on the differences | independent-samples test |
When in doubt, a rank-based test is a sound fallback: it costs only a little power if the data really were normal, and its p-value stays valid when they are not. It answers a slightly different question, though. It asks whether values in one group tend to run higher than in the other, and it compares medians only when the groups' distributions share a shape. For a question about means, Welch's t-test or a bootstrap interval is the better tool. And always pair the verdict with an effect size and a confidence interval, the lesson of the Significance, p-values & Errors chapter, so significance is never confused with importance.
Real-World Example: One Clinic Dataset, Four Questions
A clinic dataset carries numeric, paired, and categorical columns at once, the perfect workout for the decision map. We pose four questions and let the framework choose the test for each, no guessing.
One row per patient with clinic, treatment,
wait_minutes, sysbp_before, sysbp_after, and satisfied.
| Question | Data shape | Test chosen | Result | Significant? |
|---|---|---|---|---|
| Did BP drop within patients? | paired numeric | paired t-test | drop ≈ 9 mmHg, p ≈ 10⁻⁴⁸ | yes |
| New vs standard BP drop? | 2 independent numeric | two-sample (Welch) t | 11.6 vs 6.3, p ≈ 10⁻⁹ | yes |
| Satisfaction by treatment? | 2 categoricals | chi-square | p ≈ 0.27 | no |
| Wait time by clinic? | 3 groups, skewed | Kruskal-Wallis | p ≈ 0.33 (ANOVA 0.59) | no |
One spreadsheet, four questions, four correctly chosen tests, and two come back non-significant. That is exactly how real analysis goes. The framework picks the test from the data shape, and the data then answer honestly: the new treatment genuinely lowers blood pressure more than standard (and patients drop about 9 mmHg overall), while satisfaction and wait time show no reliable differences here. Reporting those null results is just as much a part of good inference as celebrating the significant ones.
Choosing Tests in Machine Learning & AI
The same decision map governs how you compare models and monitor systems.
| ML question | Data shape | Test to reach for |
|---|---|---|
| Model A vs B on the same folds | paired numeric scores | paired t (CV-corrected) / Wilcoxon signed-rank |
| Conversion of variant A vs B | two proportions | two-proportion z-test |
| Accuracy of 4 architectures on the same folds | 3+ blocked numeric scores | repeated-measures ANOVA / Friedman + post-hoc |
| Feature vs target (categorical) | two categoricals | chi-square |
| Skewed latency: A vs B | 2 groups, heavy tail | Mann-Whitney U |
Rigorous ML evaluation is mostly test selection in disguise. Comparing two models on shared folds is paired, and comparing conversion rates is a two-proportion problem. Benchmarking several models on the same folds calls for a repeated-measures test (repeated-measures ANOVA or Friedman, plus a post-hoc), and skewed metrics like latency or loss push you toward the rank-based twins. Picking the wrong test, for instance treating shared-fold scores as independent, can erase a real improvement or manufacture a fake one. Even the paired t-test overstates significance on k-fold scores, because the folds share training data, which is why Nadeau and Bengio (2003) proposed a corrected resampled t-test. The decision map is the checklist that keeps "model B is better" an honest claim.
Walk the decision map in Python
The companion notebook prints the full selection table, then routes four questions about
choosing-the-right-test--clinic_visits.xlsx, paired BP change, treatment effect, satisfaction, and wait time, to
the correct test, ending in a one-look decision summary.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Four questions pick the test: outcome type, number of groups, paired vs independent, and assumptions.
- ✓Numeric → t / ANOVA (or Mann-Whitney / Kruskal-Wallis); categorical → z for proportions / chi-square.
- ✓When assumptions fail (skew, outliers, ordinal), switch to the rank-based twin, which asks whether one group tends to run higher.
- ✓Worked example: the framework chose four tests on one clinic dataset; the new treatment won, satisfaction and wait time showed no significant difference.
- ✓In ML/AI: model comparison is test selection, shared folds are paired, rates are two-proportion, skewed metrics go rank-based.
Practice Challenges
Five short challenges, beginner to intermediate. Name the test first, then run it on the clinic data.
Numeric, 2 paired
Outcome numeric, two paired groups, roughly normal. Name the test, then run it on BP before vs after.
Numeric, 2 independent
Name the test for two independent numeric groups; run it on BP drop by treatment.
Two categoricals
Name the test for two categorical variables; run it on treatment × satisfied.
Numeric, 3 groups, skewed
Name the test for a skewed numeric outcome across three groups; run it on wait time by clinic.
One proportion vs a target
Is the overall satisfaction rate different from 0.70? Pick and run the test.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on choosing the right test. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Everything in this Part converges in one practice. A/B Testing & Online Experiments ties hypotheses, power, proportion tests, and honest reporting into the workflow that runs modern product decisions, and closes Hypothesis Testing & Inference.