You rarely know the population standard deviation, you estimate it from the sample. That extra uncertainty is exactly what Student's t distribution accounts for, with slightly heavier tails than the normal. The t-test is the most used test in all of applied statistics.
Three designs, three t-tests: one mean vs a target, two independent groups (use Welch), or matched pairs (analyze the differences). Picking the wrong one, especially ignoring pairing, can flip your conclusion.
The One-Sample t-Test
The one-sample t-test asks whether a single group's mean differs from a fixed value μ₀. Because we estimate σ with the sample sd s, we use Student's t rather than the normal, the price of that estimate is a little extra width in the tails.
This is the test behind the factory short-fill from the Logic of Hypothesis Testing chapter. With df = n − 1, small samples get fatter tails (so a larger t is needed to reach significance), and as n grows the t distribution slides back toward the normal, reconnecting with the z-test of the previous chapter.
The Two-Sample t-Test
To compare the means of two independent groups we test H₀: μA = μB. The modern default is Welch's t-test, which does not assume the two groups have equal variances. It is almost never wrong, so reach for it first.
Welch's test adjusts the degrees of freedom for unequal spreads; the classic pooled t-test (which assumes equal variances) is a special case you should use only with good reason. The same idea extends to a one-sided alternative when you specifically predict which group is larger.
The Paired t-Test
When the two measurements are paired, before and after on the same person, left and right hands, matched cases, they are not independent. The trick: collapse each pair into its difference and run a one-sample t-test on those differences. When the two measurements are positively correlated, as they usually are on the same person, pairing cancels the between-subject noise and makes the test much more powerful.
In the notebook the same numbers give a decisive paired result (t = −4.67, p ≈ 0.00004) but a non-significant result (p ≈ 0.22) when the pairing is wrongly ignored, the within-subject signal is drowned in between-subject variation. The design dictates the test.
Real-World Example: Did the Flipped Classroom Work?
A school recorded each student's pre and post test scores under two teaching methods. One dataset answers three different questions, each a different t-test, the perfect illustration of matching the test to the design.
One row per student with pretest, posttest, and the
method (traditional or flipped).
| Question | Test | Result | Verdict |
|---|---|---|---|
| Did students improve? | paired t (post vs pre) | gain ≈ 8.0 pts, t = 15.78 | yes, decisively |
| Flipped vs traditional? | two-sample (Welch) | 74.2 vs 65.6, t = 4.41, p ≈ 0.00002 | flipped wins |
| Gain above a 5-pt target? | one-sample t (gain vs 5) | t = 5.95, p ≈ 0.00000003 | yes, exceeds 5 |
All three are decisive, and each needed its own test. The paired test exploits that each post score has a matching pre score on the same student, isolating the ~8-point learning gain. The two-sample Welch test compares the two independent method groups (74.2 vs 65.6). The flipped group also started higher (pretest 63.5 vs 59.7), so part of that posttest gap was there before teaching began. Its gain was still larger (10.8 vs 5.9 points, Welch t = 5.36), so the verdict holds, but the gap in gains (about 4.9 points) is well under the raw 8.6. The one-sample test checks the average gain against a policy target of 5 points. Same spreadsheet, three questions, three correctly chosen t-tests.
t-Tests in Machine Learning & AI
The paired t-test in particular is a staple of rigorous model comparison.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Paired t-test | Comparing two models on the same folds | model A vs B across CV folds |
| Two-sample t-test | Comparing a metric across two cohorts | latency on region 1 vs region 2 |
| One-sample t-test | "Is mean error below a target?" | average loss vs an SLA threshold |
| Welch over pooled | Robust default for unequal variance | noisy vs stable model outputs |
When you compare two models, evaluating both on the same cross-validation folds or the same test examples makes the comparison paired, and a paired t-test on the per-fold score differences is far more sensitive than treating the two score lists as independent. Folds in k-fold cross-validation share most of their training data, though, so their differences are not independent and the plain paired t-test overstates significance. The corrected resampled t-test of Nadeau and Bengio (2003) widens the standard error to allow for that. The same caveats from the Significance, p-values & Errors chapter hold: report the effect size (the mean improvement), watch for multiple comparisons across many benchmarks, and remember that with skewed or heavy-tailed metrics a nonparametric test (see Nonparametric Tests) may be safer.
Run all three t-tests in Python
The companion notebook builds the one-sample t by hand and with SciPy, contrasts Welch and pooled
two-sample tests, shows paired vs unpaired on the same data, and loads
t-tests--class_scores.xlsx to run all three tests on the classroom study.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓t-test: like z but with Student's t (df = n − 1 for one sample), the right tool when σ is unknown and n is modest, provided the observations are independent and roughly normal (which matters less as n grows).
- ✓One-sample tests a mean vs a value; two-sample compares two independent groups (use Welch).
- ✓Paired analyzes within-pair differences; ignoring the pairing can hide a real effect.
- ✓Worked example: students gained ~8 pts (paired), flipped beat traditional (Welch, p ≈ 0.00002), gain exceeded a 5-pt target.
- ✓In ML/AI: compare two models on the same folds with a paired t-test on per-fold differences.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
One-sample t-test
For a sample of 30 from N(48, 10), test H₀: μ = 50 (two-sided).
stats.ttest_1samp(x, 50).Welch two-sample t-test
Compare N(50, 8, n = 40) and N(54, 14, n = 45) with Welch's test.
equal_var=False.Paired vs unpaired
Make before ~ N(100, 15, n = 30) and after = before − N(4, 6). Compare paired and unpaired p-values.
ttest_rel vs ttest_ind.One-sided two-sample
Test H₁: μB > μA (one-sided) for two groups; halve the p-value only if the direction matches.
The class-score tests
Load t-tests--class_scores.xlsx; run the paired, two-sample (flipped vs traditional), and one-sample (gain vs 5) tests.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on t-tests. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
The two-sample t-test compares exactly two groups. ANOVA extends the idea to three or more groups at once, while keeping the overall error rate under control.