The hardest part of testing is rarely the arithmetic, the computer does that. It is choosing which test fits the question and the data. Get that right and the rest is a function call; get it wrong and even a perfect calculation gives a meaningless answer.
Answer four questions, read the test off the map, check the assumptions. The framework chooses the test; the data decide the verdict, including the perfectly valid "no significant difference".
The Four Questions
Selecting a test is mechanical once you interrogate the problem. Ask, in order:
- Outcome type? Numeric (a measurement) or categorical (a label / yes-no)?
- How many groups? One vs a benchmark, two groups, or three or more?
- Independent or paired? Different subjects in each group, or the same subjects measured twice?
- Do assumptions hold? Roughly normal (or large n) and adequate expected counts, or skewed/ordinal (use the rank-based twin)?
Those four answers pin down the test almost uniquely. The whole point of Chapters 74-80 was to stock the toolbox; this chapter is the index that tells you which tool to grab.
The Decision Map
Here is the entire toolbox on one page. Find your row by outcome type, group count, and design; the rightmost column names the test (and its rank-based fallback when assumptions fail).
| Outcome | Groups | Design | Test (rank-based fallback) |
|---|---|---|---|
| Numeric | 1 vs a value | — | one-sample t-test |
| Numeric | 2 | independent | two-sample (Welch) t-test (Mann-Whitney) |
| Numeric | 2 | paired | paired t-test (Wilcoxon signed-rank) |
| Numeric | 3+ | independent | one-way ANOVA (Kruskal-Wallis) |
| Proportion | 1 | — | one-proportion z-test |
| Proportion | 2 | independent | two-proportion z-test (chi-square) |
| Categorical | 2 variables | — | chi-square test of independence |
Assumptions & Robustness
The fourth question, assumptions, is what splits a row between its parametric test and its rank-based twin. A quick checklist before you commit:
| Check | If it holds | If it fails |
|---|---|---|
| Roughly normal (or n ≥ ~30)? | t-test / ANOVA | Mann-Whitney / Kruskal-Wallis |
| Heavy skew or outliers? | mean-based test OK | rank-based test (robust) |
| Expected counts ≥ ~5 per cell? | chi-square | exact test (e.g. Fisher's) |
| Data paired? | paired test (more power) | do not treat as independent |
When in doubt, the rank-based test is the safe default: it costs only a little power if the data really were normal, and stays valid when they are not. And always pair the verdict with an effect size and a confidence interval, the lesson of the Significance, p-values & Errors chapter, so significance is never confused with importance.
Real-World Example: One Clinic Dataset, Four Questions
A clinic dataset carries numeric, paired, and categorical columns at once, the perfect workout for the decision map. We pose four questions and let the framework choose the test for each, no guessing.
One row per patient with clinic, treatment,
wait_minutes, sysbp_before, sysbp_after, and satisfied.
| Question | Data shape | Test chosen | Result | Significant? |
|---|---|---|---|---|
| Did BP drop within patients? | paired numeric | paired t-test | drop ≈ 9 mmHg, p ≈ 10⁻⁴⁸ | yes |
| New vs standard BP drop? | 2 independent numeric | two-sample (Welch) t | 11.6 vs 6.3, p ≈ 10⁻⁹ | yes |
| Satisfaction by treatment? | 2 categoricals | chi-square | p ≈ 0.27 | no |
| Wait time by clinic? | 3 groups numeric | ANOVA / Kruskal | p ≈ 0.59 | no |
One spreadsheet, four questions, four correctly chosen tests, and two come back non-significant. That is exactly how real analysis goes. The framework picks the test from the data shape, and the data then answer honestly: the new treatment genuinely lowers blood pressure more than standard (and patients drop about 9 mmHg overall), while satisfaction and wait time show no reliable differences here. Reporting those null results is just as much a part of good inference as celebrating the significant ones.
Choosing Tests in Machine Learning & AI
The same decision map governs how you compare models and monitor systems.
| ML question | Data shape | Test to reach for |
|---|---|---|
| Model A vs B on the same folds | paired numeric scores | paired t / Wilcoxon signed-rank |
| Conversion of variant A vs B | two proportions | two-proportion z-test |
| Accuracy across 4 architectures | 3+ numeric groups | ANOVA / Kruskal + post-hoc |
| Feature vs target (categorical) | two categoricals | chi-square |
| Skewed latency: A vs B | 2 groups, heavy tail | Mann-Whitney U |
Rigorous ML evaluation is mostly test selection in disguise. Comparing two models on shared folds is paired; comparing conversion rates is a two-proportion problem; benchmarking several models is an ANOVA-plus-post-hoc problem; and skewed metrics like latency or loss push you toward the rank-based twins. Picking the wrong test, for instance treating shared-fold scores as independent, can erase a real improvement or manufacture a fake one. The decision map is the checklist that keeps "model B is better" an honest claim.
Walk the decision map in Python
The companion notebook prints the full selection table, then routes four questions about
choosing-the-right-test--clinic_visits.xlsx, paired BP change, treatment effect, satisfaction, and wait time, to
the correct test, ending in a one-look decision summary.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Four questions pick the test: outcome type, number of groups, paired vs independent, and assumptions.
- ✓Numeric → t / ANOVA (or Mann-Whitney / Kruskal-Wallis); categorical → z for proportions / chi-square.
- ✓When assumptions fail (skew, outliers, ordinal), switch to the rank-based twin, the safe default.
- ✓Real data: the framework chose four tests on one clinic dataset; the new treatment won, satisfaction and wait time did not differ.
- ✓In ML/AI: model comparison is test selection, shared folds are paired, rates are two-proportion, skewed metrics go rank-based.
Practice Challenges
Five short challenges, beginner to intermediate. Name the test first, then run it on the clinic data.
Numeric, 2 paired
Outcome numeric, two paired groups, roughly normal. Name the test, then run it on BP before vs after.
Numeric, 2 independent
Name the test for two independent numeric groups; run it on BP drop by treatment.
Two categoricals
Name the test for two categorical variables; run it on treatment × satisfied.
Numeric, 3 groups, skewed
Name the test for a skewed numeric outcome across three groups; run it on wait time by clinic.
One proportion vs a target
Is the overall satisfaction rate different from 0.70? Pick and run the test.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on choosing the right test. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Everything in this Part converges in one practice. A/B Testing & Online Experiments ties hypotheses, power, proportion tests, and honest reporting into the workflow that runs modern product decisions, and closes Hypothesis Testing & Inference.