Chapter 79 · Hypothesis Testing & Inference
t-Tests 📏
Real samples are modest and sigma is unknown, exactly where Student's t shines. We build the one-sample, two-sample (Welch), and paired t-tests, see why pairing is so powerful, then answer three questions about a flipped-classroom study with the right test for each.
Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026
⚙️ Setup¶
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
# statsmodels = the R/SAS-style stats library (pre-installed on Colab): it computes the
# standard errors, test statistics, intervals, and post-hoc comparisons, so we write less by hand.
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep, proportion_confint
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
from scipy.stats.contingency import association # Cramer's V in one call
IND="#4f46e5"; DEEP="#4338ca"; LIGHT="#818cf8"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"; AMBER="#d97706"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(77)
DEMO 1 · THE ONE-SAMPLE t-TEST
Test a mean when sigma is unknown
When the population sigma is unknown (almost always) we estimate it with the sample sd s and compare to Student's t with n-1 degrees of freedom. The statistic is t = (xbar - mu0)/(s/sqrt(n)).
In [2]:
x = rng.normal(52, 9, 25)
res = stats.ttest_1samp(x, 50)
n=len(x); t=(x.mean()-50)/(x.std(ddof=1)/np.sqrt(n))
print(f"n={n}, mean={x.mean():.2f}, sd={x.std(ddof=1):.2f}")
print(f"t (by hand) = {t:.3f}, df={n-1}")
print(f"scipy: t={res.statistic:.3f}, two-sided p={res.pvalue:.4f}")
n=25, mean=49.87, sd=10.82 t (by hand) = -0.059, df=24 scipy: t=-0.059, two-sided p=0.9538
This is the test we met in Chapter 76. With a modest sample and unknown sigma, the t distribution's heavier tails give the honest, slightly wider, accounting of uncertainty that the normal curve would understate.
DEMO 2 · THE TWO-SAMPLE t-TEST
Comparing two independent groups
To compare the means of two independent groups we test H0: muA = muB. By default use WELCH'S t-test, which does NOT assume equal variances; it is safer than the classic pooled test and rarely wrong. scipy: equal_var=False.
In [3]:
a = rng.normal(50, 8, 60) # group A
b = rng.normal(55, 12, 70) # group B (different mean AND spread)
welch = stats.ttest_ind(a, b, equal_var=False)
pooled = stats.ttest_ind(a, b, equal_var=True)
print(f"meanA={a.mean():.2f}, meanB={b.mean():.2f}, difference={b.mean()-a.mean():+.2f}")
print(f"Welch t={welch.statistic:.3f}, p={welch.pvalue:.4f} (default, unequal variances OK)")
print(f"pooled t={pooled.statistic:.3f}, p={pooled.pvalue:.4f} (assumes equal variances)")
meanA=50.93, meanB=53.24, difference=+2.31 Welch t=-1.359, p=0.1767 (default, unequal variances OK) pooled t=-1.309, p=0.1930 (assumes equal variances)
Welch's test adjusts the degrees of freedom for unequal spreads, so it stays valid when the two groups have different variances, the common case. Reach for the pooled version only when you have a real reason to believe the variances match.
DEMO 3 · THE PAIRED t-TEST
When each observation has a partner
If the two measurements are PAIRED, before/after on the same subject, left/right, matched cases, do NOT treat them as independent. Form each pair's difference and run a ONE-sample t-test on the differences. When the paired measurements are positively correlated, as they usually are, pairing removes between-subject noise, so it is far more powerful.
In [4]:
before = rng.normal(140, 16, 40)
after = before - rng.normal(6, 7, 40) # a real within-subject drop
paired = stats.ttest_rel(after, before)
wrong = stats.ttest_ind(after, before, equal_var=False) # ignoring the pairing
d = after - before
print(f"mean difference (after - before) = {d.mean():+.2f}")
print(f"PAIRED t-test: t={paired.statistic:.2f}, p={paired.pvalue:.2e} (correct)")
print(f"unpaired (WRONG): t={wrong.statistic:.2f}, p={wrong.pvalue:.3f} (throws away the pairing)")
mean difference (after - before) = -4.85 PAIRED t-test: t=-4.67, p=3.52e-05 (correct) unpaired (WRONG): t=-1.24, p=0.219 (throws away the pairing)
Same data, very different verdicts: the paired test sees the consistent within-person drop and is decisive, while wrongly treating the columns as independent buries that signal in between-person variation. Match the test to the design, paired data needs a paired test.
REAL-WORLD EXAMPLE · DID THE FLIPPED CLASSROOM WORK?
Three t-tests on one class dataset
A school tested two teaching methods and recorded each student's pre and post scores (t-tests--class_scores.xlsx). We ask three questions, each a different t-test: did students improve (paired), did the flipped method beat traditional (two-sample), and did the average gain exceed a 5-point target (one-sample)?
In [5]:
try: sc = pd.read_excel("../../data/t-tests--class_scores.xlsx", sheet_name="Scores")
except FileNotFoundError: sc = pd.read_excel(BASE+"t-tests--class_scores.xlsx", sheet_name="Scores")
# EXPLORE FIRST: size, missing, and pre/post means by method
print("shape:", sc.shape, "| missing:", sc.isna().sum().sum())
print(sc.groupby("method")[["pretest","posttest"]].agg(["size","mean"]).round(1))
gain = sc.posttest - sc.pretest
paired = stats.ttest_rel(sc.posttest, sc.pretest)
print(f"[PAIRED] mean gain = {gain.mean():.2f} pts, t={paired.statistic:.2f}, p={paired.pvalue:.2e} -> students improved")
shape: (120, 4) | missing: 0
pretest posttest
size mean size mean
method
flipped 53 63.5 53 74.2
traditional 67 59.7 67 65.6
[PAIRED] mean gain = 8.03 pts, t=15.78, p=6.00e-31 -> students improved
In [6]:
fl = sc[sc.method=="flipped"].posttest; tr = sc[sc.method=="traditional"].posttest
two = stats.ttest_ind(fl, tr, equal_var=False)
print(f"[TWO-SAMPLE] flipped {fl.mean():.2f} vs traditional {tr.mean():.2f} on posttest")
print(f" Welch t={two.statistic:.2f}, p={two.pvalue:.2e} -> flipped is better")
nfl, ntr = len(fl), len(tr)
sp = np.sqrt(((nfl-1)*fl.std(ddof=1)**2 + (ntr-1)*tr.std(ddof=1)**2)/(nfl+ntr-2)) # pooled SD
print(f" Cohen d = {(fl.mean()-tr.mean())/sp:.2f} (effect size; 0.2 small, 0.5 medium, 0.8 large)")
gf, gt = gain[sc.method=="flipped"], gain[sc.method=="traditional"] # flipped started higher, so compare gains too
gw = stats.ttest_ind(gf, gt, equal_var=False)
print(f" gains: flipped {gf.mean():.2f} vs traditional {gt.mean():.2f}, Welch t={gw.statistic:.2f}, p={gw.pvalue:.1e}")
one = stats.ttest_1samp(gain, 5)
print(f"[ONE-SAMPLE] mean gain vs target 5: t={one.statistic:.2f}, p={one.pvalue:.2e} -> gain exceeds 5")
fig,ax=plt.subplots(1,2,figsize=(11,3.3))
ax[0].scatter(sc.pretest, sc.posttest, c=(sc.method=="flipped").map({True:IND,False:LIGHT}), alpha=0.7, s=22)
lims=[sc.pretest.min(),sc.posttest.max()]; ax[0].plot(lims,lims,color=INK,ls="--",lw=1)
ax[0].set_xlabel("pretest"); ax[0].set_ylabel("posttest"); ax[0].set_title("Post vs pre (above line = improved)")
ax[1].boxplot([tr,fl], tick_labels=["traditional","flipped"]); ax[1].set_ylabel("posttest")
ax[1].set_title("Posttest by method"); plt.tight_layout(); plt.show()
[TWO-SAMPLE] flipped 74.22 vs traditional 65.61 on posttest
Welch t=4.41, p=2.33e-05 -> flipped is better
Cohen d = 0.81 (effect size; 0.2 small, 0.5 medium, 0.8 large)
gains: flipped 10.76 vs traditional 5.87, Welch t=5.36, p=4.2e-07
[ONE-SAMPLE] mean gain vs target 5: t=5.95, p=2.71e-08 -> gain exceeds 5
All three tests are decisive. Students improved by about 8 points on average (paired t = 15.8, p tiny). The flipped method scored higher on the posttest (74.2 vs 65.6, Welch t = 4.41, p ≈ 0.00002), though it also started higher (pretest 63.5 vs 59.7), and its gain was still larger (10.8 vs 5.9 points, Welch t = 5.36). The average gain comfortably beat the 5-point target (one-sample t = 5.95). The lesson is choosing the right t-test for each question: one design, three tests.
✅ t-tests
The t-test handles an unknown sigma via Student's t (n-1 df for one sample or for paired differences). One-sample tests a mean against a value; the two-sample (Welch by default) compares two independent groups; the paired test analyzes within-pair differences and is far more powerful when the paired measurements are positively correlated. On the teaching class data, students gained ~8 points (paired), the flipped method scored higher (two-sample, p≈0.00002, with a larger gain from a higher start), and the gain beat a 5-point target. Next: ANOVA, the t-test extended to three or more groups.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher