Chapter 83 · Hypothesis Testing & Inference
Choosing the Right Test 🧭
You now own a toolbox of tests, the skill is picking the right one. Four questions (outcome type, number of groups, paired or not, assumptions) route every problem to its test. We run one mixed clinic dataset through the whole decision map.
Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026
⚙️ Setup¶
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
# statsmodels = the R/SAS-style stats library (pre-installed on Colab): it computes the
# standard errors, test statistics, intervals, and post-hoc comparisons, so we write less by hand.
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep, proportion_confint
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
from scipy.stats.contingency import association # Cramer's V in one call
IND="#4f46e5"; DEEP="#4338ca"; LIGHT="#818cf8"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"; AMBER="#d97706"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(81)
DEMO 1 · THE FOUR QUESTIONS
A decision tree, not a memory test
Choosing a test is mechanical once you answer four questions: (1) Is the OUTCOME numeric or categorical? (2) How many GROUPS are you comparing (one, two, 3+)? (3) Are the groups INDEPENDENT or PAIRED? (4) Do the ASSUMPTIONS (rough normality, expected counts >= 5) hold, or do you need a rank-based test?
In [2]:
guide = pd.DataFrame([
["numeric","1","-","one-sample t-test"],
["numeric","2","independent","two-sample (Welch) t-test / Mann-Whitney if skewed"],
["numeric","2","paired","paired t-test / Wilcoxon signed-rank if skewed"],
["numeric","3+","independent","one-way ANOVA / Kruskal-Wallis if skewed"],
["proportion","1","-","one-proportion z-test"],
["proportion","2","independent","two-proportion z-test / chi-square"],
["categorical","2+ vars","-","chi-square test of independence"],
], columns=["outcome","groups","design","test"])
print(guide.to_string(index=False))
outcome groups design test
numeric 1 - one-sample t-test
numeric 2 independent two-sample (Welch) t-test / Mann-Whitney if skewed
numeric 2 paired paired t-test / Wilcoxon signed-rank if skewed
numeric 3+ independent one-way ANOVA / Kruskal-Wallis if skewed
proportion 1 - one-proportion z-test
proportion 2 independent two-proportion z-test / chi-square
categorical 2+ vars - chi-square test of independence
That is the entire chapter in one table. The rest is practice: read the question, identify the outcome type and the group structure, check the assumptions, and the test picks itself. We will run the clinic dataset through it.
DEMO 2 · NUMERIC OUTCOMES
Paired vs two-group vs many-group
Load the clinic data. Numeric outcomes split three ways. Before/after on the SAME patient is PAIRED (paired t). Two independent arms is a two-sample t. Three clinics is ANOVA. Skew pushes each toward its rank-based twin.
In [3]:
try: d = pd.read_excel("../../data/choosing-the-right-test--clinic_visits.xlsx", sheet_name="Visits")
except FileNotFoundError: d = pd.read_excel(BASE+"choosing-the-right-test--clinic_visits.xlsx", sheet_name="Visits")
# EXPLORE FIRST: size, missing, and a quick summary of the numeric columns
print("shape:", d.shape, "| missing:", d.isna().sum().sum())
print(d[["wait_minutes","sysbp_before","sysbp_after"]].describe().T[["mean","std","min","max"]].round(1))
print("treatment:", d.treatment.value_counts().to_dict(), "| satisfied rate:", round(d.satisfied.mean(),3))
# Q: did blood pressure drop within patients? PAIRED numeric -> paired t-test
pt = stats.ttest_rel(d.sysbp_after, d.sysbp_before)
print(f"[paired t] mean BP drop = {(d.sysbp_before-d.sysbp_after).mean():.2f} mmHg, t={pt.statistic:.2f}, p={pt.pvalue:.2e}")
# Q: does the NEW treatment drop BP more than standard? TWO independent groups -> Welch t
drop=d.sysbp_before-d.sysbp_after
new=drop[d.treatment=="new"]; std=drop[d.treatment=="standard"]
tt=stats.ttest_ind(new,std,equal_var=False)
print(f"[two-sample t] new drop {new.mean():.2f} vs standard {std.mean():.2f}, t={tt.statistic:.2f}, p={tt.pvalue:.2e}")
shape: (200, 7) | missing: 0
mean std min max
wait_minutes 22.8 12.1 3.6 57.9
sysbp_before 147.7 11.8 110.0 175.0
sysbp_after 138.8 13.8 103.0 173.0
treatment: {'new': 102, 'standard': 98} | satisfied rate: 0.685
[paired t] mean BP drop = 8.96 mmHg, t=-19.54, p=3.68e-48
[two-sample t] new drop 11.57 vs standard 6.26, t=6.35, p=1.49e-09
In [4]:
fig,ax=plt.subplots(1,2,figsize=(11,3.5))
# LEFT: PAIRED -> each patient before->after on the same line (a paired design)
samp=d.sample(35, random_state=1)
for _,r in samp.iterrows(): ax[0].plot([0,1],[r.sysbp_before,r.sysbp_after],color=LIGHT,alpha=0.45,lw=1)
ax[0].plot([0,1],[d.sysbp_before.mean(),d.sysbp_after.mean()],color=DEEP,lw=3,marker="o",ms=7,label="mean")
ax[0].set_xlim(-0.25,1.25); ax[0].set_xticks([0,1]); ax[0].set_xticklabels(["before","after"])
ax[0].set_ylabel("systolic BP (mmHg)"); ax[0].set_title("PAIRED numeric -> paired t-test"); ax[0].legend()
# RIGHT: TWO INDEPENDENT GROUPS -> compare the BP drop across treatment arms
drop=d.sysbp_before-d.sysbp_after
bp=ax[1].boxplot([drop[d.treatment=="standard"], drop[d.treatment=="new"]], tick_labels=["standard","new"], patch_artist=True)
for patch,c in zip(bp["boxes"],[LIGHT,IND]): patch.set_facecolor(c); patch.set_alpha(0.6)
ax[1].set_ylabel("BP drop (mmHg)"); ax[1].set_title("2 INDEPENDENT groups -> two-sample t")
plt.tight_layout(); plt.show()
Same dataset, two numeric questions, two different t-tests, chosen purely from the design: within-patient change is paired (left, the connected lines), while the between-treatment comparison is independent two-sample (right, separate boxes). Misreading paired data as independent (Chapter 79) is the classic mistake this framework prevents.
DEMO 3 · CATEGORICAL & MANY-GROUP
Proportions, chi-square, and ANOVA
A categorical outcome (satisfied yes/no by treatment) is a chi-square / two-proportion question. A numeric outcome across 3+ groups (wait time by clinic) is ANOVA, or Kruskal-Wallis if skewed. The framework routes each automatically.
In [5]:
# Q: is satisfaction associated with treatment? TWO categoricals -> chi-square
ct=pd.crosstab(d.treatment, d.satisfied)
chi2,p,dof,_=stats.chi2_contingency(ct)
print(f"[chi-square] treatment x satisfied: chi2={chi2:.2f}, dof={dof}, p={p:.3f} -> {'associated' if p<0.05 else 'no clear association'}")
# Q: does wait time differ by clinic? 3 groups numeric (skewed) -> ANOVA & Kruskal
groups=[g.wait_minutes.values for _,g in d.groupby("clinic")]
F,pf=stats.f_oneway(*groups); H,pk=stats.kruskal(*groups)
print(f"[ANOVA] wait by clinic: F={F:.2f}, p={pf:.3f}")
print(f"[Kruskal] wait by clinic: H={H:.2f}, p={pk:.3f} (skewed -> trust the rank test)")
[chi-square] treatment x satisfied: chi2=1.22, dof=1, p=0.269 -> no clear association [ANOVA] wait by clinic: F=0.53, p=0.588 [Kruskal] wait by clinic: H=2.20, p=0.333 (skewed -> trust the rank test)
In [6]:
fig,ax=plt.subplots(1,2,figsize=(11,3.5))
# LEFT: TWO CATEGORICALS -> compare a rate across categories
rates=d.groupby("treatment").satisfied.mean()
ax[0].bar(rates.index, rates.values, color=[LIGHT,IND], width=0.55)
for i,v in enumerate(rates.values): ax[0].text(i,v+0.012,f"{v:.0%}",ha="center",fontweight="bold")
ax[0].set_ylim(0,1); ax[0].set_ylabel("share satisfied"); ax[0].set_title("2 CATEGORICALS -> chi-square")
# RIGHT: 3 GROUPS, skewed numeric -> ANOVA / Kruskal
order=sorted(d.clinic.unique())
bp=ax[1].boxplot([d[d.clinic==c].wait_minutes.values for c in order], tick_labels=order, patch_artist=True)
for patch in bp["boxes"]: patch.set_facecolor(IND); patch.set_alpha(0.55)
ax[1].set_ylabel("wait (minutes)"); ax[1].set_title("3 GROUPS, skewed -> ANOVA / Kruskal")
plt.tight_layout(); plt.show()
Not every test fires: satisfaction is not clearly tied to treatment here (the two bars are close, p ≈ 0.27), and wait time shows no significant difference by clinic (the three boxes overlap heavily, Kruskal p ≈ 0.33). That is a feature, not a failure, the framework tells you which test to run; the data tell you the answer, and "no significant difference" is a legitimate, common result.
REAL-WORLD EXAMPLE · ONE CLINIC DATASET, FOUR QUESTIONS
The decision map, applied end to end
The clinic dataset (choosing-the-right-test--clinic_visits.xlsx) carries numeric, paired, and categorical columns at once, the perfect workout for the decision map. We pose four questions and let the framework choose the test for each.
In [7]:
rows=[]
rows.append(("Did BP drop within patients?","paired numeric","paired t-test",f"p={stats.ttest_rel(d.sysbp_after,d.sysbp_before).pvalue:.1e}","YES"))
rows.append(("New vs standard BP drop?","2 indep. numeric","two-sample (Welch) t",f"p={stats.ttest_ind(new,std,equal_var=False).pvalue:.1e}","YES"))
rows.append(("Satisfaction by treatment?","2 categoricals","chi-square",f"p={stats.chi2_contingency(pd.crosstab(d.treatment,d.satisfied))[1]:.2f}","no"))
rows.append(("Wait time by clinic?","3 groups, skewed","Kruskal-Wallis",f"p={stats.kruskal(*groups).pvalue:.2f}","no"))
res=pd.DataFrame(rows, columns=["question","data shape","test chosen","result","significant?"])
print(res.to_string(index=False))
question data shape test chosen result significant?
Did BP drop within patients? paired numeric paired t-test p=3.7e-48 YES
New vs standard BP drop? 2 indep. numeric two-sample (Welch) t p=1.5e-09 YES
Satisfaction by treatment? 2 categoricals chi-square p=0.27 no
Wait time by clinic? 3 groups, skewed Kruskal-Wallis p=0.33 no
One spreadsheet, four questions, four correctly chosen tests, and two of them come back non-significant. That is exactly how real analysis goes: the hard part is rarely the arithmetic, it is matching the test to the question and the data shape, and then reporting honestly whether the effect is there. The new treatment genuinely lowers blood pressure more than standard; satisfaction and wait time show no reliable differences here.
✅ Choosing the right test
Four questions pick the test: outcome type (numeric vs categorical), number of groups (1 / 2 / 3+), design (independent vs paired), and whether assumptions hold (else go nonparametric). Numeric → t / ANOVA (or Mann-Whitney / Kruskal-Wallis); categorical → z for proportions / chi-square. On one clinic dataset the new treatment beats standard on BP drop, while satisfaction and wait time show no reliable difference. Next, the capstone of the Part: A/B testing and online experiments.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher