You rarely know the population standard deviation, you estimate it from the sample. That extra uncertainty is exactly what Student's t distribution accounts for, with slightly heavier tails than the normal. The t-test is the most used test in all of applied statistics.
Three designs, three t-tests: one mean vs a target, two independent groups (use Welch), or matched pairs (analyze the differences). Picking the wrong one, especially ignoring pairing, can flip your conclusion.
The One-Sample t-Test
The one-sample t-test asks whether a single group's mean differs from a fixed value μ₀. Because we estimate σ with the sample sd s, we use Student's t rather than the normal, the price of that estimate is a little extra width in the tails.
This is the test behind the factory short-fill from the Logic of Hypothesis Testing chapter. With df = n − 1, small samples get fatter tails (so a larger t is needed to reach significance), and as n grows the t distribution slides back toward the normal, reconnecting with the z-test of the previous chapter.
The Two-Sample t-Test
To compare the means of two independent groups we test H₀: μ₀ = μ₊. The modern default is Welch's t-test, which does not assume the two groups have equal variances. It is almost never wrong, so reach for it first.
Welch's test adjusts the degrees of freedom for unequal spreads; the classic pooled t-test (which assumes equal variances) is a special case you should use only with good reason. The same idea extends to a one-sided alternative when you specifically predict which group is larger.
The Paired t-Test
When the two measurements are paired, before and after on the same person, left and right hands, matched cases, they are not independent. The trick: collapse each pair into its difference and run a one-sample t-test on those differences. Pairing cancels the between-subject noise, making the test much more powerful.
In the notebook the same numbers give a decisive paired result (t = −4.67, p ≈ 0.00004) but a non-significant result (p ≈ 0.22) when the pairing is wrongly ignored, the within-subject signal is drowned in between-subject variation. The design dictates the test.
Real-World Example: Did the Flipped Classroom Work?
A school recorded each student's pre and post test scores under two teaching methods. One dataset answers three different questions, each a different t-test, the perfect illustration of matching the test to the design.
One row per student with pretest, posttest, and the
method (traditional or flipped).
| Question | Test | Result | Verdict |
|---|---|---|---|
| Did students improve? | paired t (post vs pre) | gain ≈ 8.0 pts, t = 15.78 | yes, decisively |
| Flipped vs traditional? | two-sample (Welch) | 74.2 vs 65.6, t = 4.41, p ≈ 0.00002 | flipped wins |
| Gain above a 5-pt target? | one-sample t (gain vs 5) | t = 5.95, p ≈ 0.00000003 | yes, exceeds 5 |
All three are decisive, and each needed its own test. The paired test exploits that each post score has a matching pre score on the same student, isolating the ~8-point learning gain. The two-sample Welch test compares the two independent method groups (74.2 vs 65.6). The one-sample test checks the average gain against a policy target of 5 points. Same spreadsheet, three questions, three correctly chosen t-tests.
t-Tests in Machine Learning & AI
The paired t-test in particular is a staple of rigorous model comparison.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Paired t-test | Comparing two models on the same folds | model A vs B across CV folds |
| Two-sample t-test | Comparing a metric across two cohorts | latency on region 1 vs region 2 |
| One-sample t-test | "Is mean error below a target?" | average loss vs an SLA threshold |
| Welch over pooled | Robust default for unequal variance | noisy vs stable model outputs |
When you compare two models, evaluating both on the same cross-validation folds or the same test examples makes the comparison paired, and a paired t-test on the per-fold score differences is far more sensitive than treating the two score lists as independent. This is the standard way to claim "model B beats model A" with statistical backing. The same caveats from the Significance, p-values & Errors chapter hold: report the effect size (the mean improvement), watch for multiple comparisons across many benchmarks, and remember that with skewed or heavy-tailed metrics a nonparametric test (see Nonparametric Tests) may be safer.
Run all three t-tests in Python
The companion notebook builds the one-sample t by hand and with SciPy, contrasts Welch and pooled
two-sample tests, shows paired vs unpaired on the same data, and loads
t-tests--class_scores.xlsx to run all three tests on the classroom study.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓t-test: like z but with Student's t (df = n − 1), the right tool when σ is unknown and n is modest.
- ✓One-sample tests a mean vs a value; two-sample compares two independent groups (use Welch).
- ✓Paired analyzes within-pair differences; ignoring the pairing can hide a real effect.
- ✓Real data: students gained ~8 pts (paired), flipped beat traditional (Welch, p ≈ 0.00002), gain exceeded a 5-pt target.
- ✓In ML/AI: compare two models on the same folds with a paired t-test on per-fold differences.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
One-sample t-test
For a sample of 30 from N(48, 10), test H₀: μ = 50 (two-sided).
stats.ttest_1samp(x, 50).Welch two-sample t-test
Compare N(50, 8, n = 40) and N(54, 14, n = 45) with Welch's test.
equal_var=False.Paired vs unpaired
Make before ~ N(100, 15, n = 30) and after = before − N(4, 6). Compare paired and unpaired p-values.
ttest_rel vs ttest_ind.One-sided two-sample
Test H₁: μ₊ > μ₀ (one-sided) for two groups; halve the p-value only if the direction matches.
Real data: three class tests
Load t-tests--class_scores.xlsx; run the paired, two-sample (flipped vs traditional), and one-sample (gain vs 5) tests.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on t-tests. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
The two-sample t-test compares exactly two groups. ANOVA extends the idea to three or more groups at once, while keeping the overall error rate under control.