Chapter 85 · Inference Case Studies
Review & Choosing the Right Test 🧭
Before the case studies, one page that ties estimation and testing together: the master which-test decision rule as runnable code, the assumption checks that come first, and the habit of reporting an effect size and interval alongside every p-value.
Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026
⚙️ Setup¶
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
# statsmodels = the R/SAS-style stats library (pre-installed on Colab): it does the standard errors,
# test statistics, intervals, and post-hoc comparisons for us, so we write far less by-hand code.
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep, proportion_confint
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
from scipy.stats.contingency import association # Cramer's V in one call
CY="#0891b2"; DEEP="#0e7490"; LIGHT="#67e8f9"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"; AMBER="#d97706"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
pd.set_option("display.width",120)
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(83)
STEP 0 · ALWAYS EXPLORE FIRST
Descriptive statistics before any test
Before a single p-value, a statistician explores: shape, spread, missing values, group sizes, outliers. The right test depends on what you see. Every case study in Chapters 86-90 opens with this step; treat .describe() and a quick plot as non-negotiable.
In [2]:
x = rng.normal(3.2, 9, 60) # e.g. a change score
s = pd.Series(x)
print(s.describe().round(2).to_dict())
print(f"skewness = {stats.skew(x):.2f} (moderate left skew, but n = {len(x)} is large enough for the t-tools)")
{'count': 60.0, 'mean': 3.18, 'std': 8.07, 'min': -21.21, '25%': -1.65, '50%': 5.06, '75%': 9.55, 'max': 17.0}
skewness = -0.62 (moderate left skew, but n = 60 is large enough for the t-tools)
STEP 1 · THE TWO SIDES OF INFERENCE
Estimation and testing answer the same data, two ways
Estimation asks 'what is the value, and how sure are we?' (a point estimate + a confidence interval). Testing asks 'is the effect real?' (H0 vs H1 + a p-value). They are mirror images: if a 95% CI for a difference excludes 0, the matching test rejects H0 at 5%. We let statsmodels' DescrStatsW give the interval, no hand-rolled standard error.
In [3]:
ds = DescrStatsW(x)
lo,hi = ds.tconfint_mean() # 95% CI from statsmodels
t,p,_ = ds.ttest_mean(0) # one-sample t-test vs 0
print(f"ESTIMATION 95% CI for the mean: [{lo:.2f}, {hi:.2f}]")
print(f"TESTING H0: mean=0 -> t={t:.2f}, p={p:.4f}")
print(f"agree? CI excludes 0 = {lo>0 or hi<0}; test rejects at 5% = {p<0.05}")
ESTIMATION 95% CI for the mean: [1.10, 5.27] TESTING H0: mean=0 -> t=3.05, p=0.0034 agree? CI excludes 0 = True; test rejects at 5% = True
Always report both: the test says whether there is an effect, the interval says how big it plausibly is. A p-value alone is half an answer. (Throughout this Part we use scipy and statsmodels, the Python equivalents of R and SAS, so the libraries compute the standard errors and test statistics for us.)
STEP 2 · THE MASTER DECISION RULE
Four questions pick the test
Outcome type (numeric vs categorical), number of groups (1 / 2 / 3+), design (independent vs paired), and assumptions (normal-ish vs skewed) determine the test almost uniquely. Here it is as a function you can actually call.
In [4]:
def choose_test(outcome, groups, design="independent", normal=True):
if outcome=="proportion":
return "one-proportion z-test" if groups==1 else "two-proportion z-test / chi-square"
if outcome=="categorical":
return "chi-square test of independence"
# numeric outcome
if groups==1: return "one-sample t-test" if normal else "Wilcoxon signed-rank"
if groups==2 and design=="paired": return "paired t-test" if normal else "Wilcoxon signed-rank"
if groups==2: return "two-sample (Welch) t-test" if normal else "Mann-Whitney U"
return "one-way ANOVA (+Tukey)" if normal else "Kruskal-Wallis"
tests=[("numeric",1,"-",True),("numeric",2,"paired",True),("numeric",2,"independent",False),
("numeric",3,"independent",True),("proportion",1,"-",True),("proportion",2,"-",True),("categorical",2,"-",True)]
for o,g,dz,nm in tests: print(f"{o:>11} | {g} groups | {dz:>11} | normal={nm} -> {choose_test(o,g,dz,nm)}")
numeric | 1 groups | - | normal=True -> one-sample t-test
numeric | 2 groups | paired | normal=True -> paired t-test
numeric | 2 groups | independent | normal=False -> Mann-Whitney U
numeric | 3 groups | independent | normal=True -> one-way ANOVA (+Tukey)
proportion | 1 groups | - | normal=True -> one-proportion z-test
proportion | 2 groups | - | normal=True -> two-proportion z-test / chi-square
categorical | 2 groups | - | normal=True -> chi-square test of independence
Memorize the four questions, not the tests; the table falls out of them. The case studies in Chapters 86-90 each run this exact decision out loud before computing anything.
STEP 3 · CHECK THE ASSUMPTIONS FIRST
The fourth question is not optional
Before trusting a t-test or ANOVA, check roughly-normal (or n>=30), and for chi-square check expected counts >= ~5, with the quick programmatic checks below. If they fail, switch to a fallback (Welch, a rank-based test, Fisher's exact), keeping in mind that a rank test asks whether one group tends to run higher.
In [5]:
skewed = rng.lognormal(1.0, 0.7, 40)
print(f"skewness = {stats.skew(skewed):.2f}, n = {len(skewed)} (|skew|>1 is strong skew; at this n the CLT helps only partly, so cross-check with a rank test or a bootstrap)")
print(f"Shapiro-Wilk normality p = {stats.shapiro(skewed).pvalue:.4f} (small p -> not normal)")
tbl = np.array([[8,2],[3,7]])
chi2,p,dof,expected = stats.chi2_contingency(tbl)
print(f"chi-square min expected count = {expected.min():.1f} ({'OK' if expected.min()>=5 else 'TOO LOW -> use Fisher exact'})")
skewness = 1.41, n = 40 (|skew|>1 is strong skew; at this n the CLT helps only partly, so cross-check with a rank test or a bootstrap) Shapiro-Wilk normality p = 0.0006 (small p -> not normal) chi-square min expected count = 4.5 (TOO LOW -> use Fisher exact)
A test is only as trustworthy as its assumptions. The honest workflow is check, then choose, never the reverse.
STEP 4 · READ THE RESULT HONESTLY
p-value + effect size + interval, every time
A significant p says data like these would be unusual if the effect were zero; it does NOT say the effect is large or important. Pair every p-value with an effect size and a confidence interval, and remember 'not significant' can just mean underpowered.
In [6]:
big = rng.normal(0.06, 1, 50000) # a tiny true effect, huge sample
res = stats.ttest_1samp(big, 0); d = big.mean()/big.std()
print(f"p = {res.pvalue:.2e} (very significant)")
print(f"effect size d = {d:.3f} (negligible) -> significant but not important")
print("Reporting checklist: estimate, CI, test/p, effect size, assumptions, and the practical decision.")
p = 2.84e-41 (very significant) effect size d = 0.060 (negligible) -> significant but not important Reporting checklist: estimate, CI, test/p, effect size, assumptions, and the practical decision.
That is the whole of inference in one habit: choose the right test for the data, then report the size of the effect with its uncertainty, not just whether p cleared 0.05.
📋 STATISTICIAN’S REPORT
Your roadmap through the case studies
Chapters 86-91 each take ONE realistic dataset from question to plain-English recommendation, narrating every decision. Ch86 a web A/B test (two-proportion z, power, lift CI). Ch87 comparing marketing channels (one-way ANOVA + Tukey). Ch88 a clinical trial (paired, two-arm, and responder-rate tests on one trial). Ch89 a customer survey (chi-square, a proportion with a margin of error, and an ordinal test). Ch90 manufacturing quality (one-sample vs spec, ANOVA across lines, defect-by-shift chi-square). Ch91 a paired pre-post study (a paired t-test on each user's difference, with Wilcoxon and sign-test cross-checks). Watch how the same four questions choose the tool every time.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher