←Back to the bookChapter 83 · Review & Choosing the Right Test

Chapter 83 · Inference Case Studies

Review & Choosing the Right Test 🧭

Before the case studies, one page that ties estimation and testing together: the master which-test decision rule as runnable code, the assumption checks that come first, and the habit of reporting an effect size and interval alongside every p-value.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

⚙️ Setup

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
# statsmodels = the R/SAS-style stats library (pre-installed on Colab): it does the standard errors,
# test statistics, intervals, and post-hoc comparisons for us, so we write far less by-hand code.
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep, proportion_confint
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
from scipy.stats.contingency import association   # Cramer's V in one call
CY="#0891b2"; DEEP="#0e7490"; LIGHT="#67e8f9"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"; AMBER="#d97706"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
pd.set_option("display.width",120)
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(83)

STEP 0 · ALWAYS EXPLORE FIRST

Descriptive statistics before any test

Before a single p-value, a statistician explores: shape, spread, missing values, group sizes, outliers. The right test depends on what you see. Every case study in Chapters 84-88 opens with this step; treat .describe() and a quick plot as non-negotiable.

x = rng.normal(3.2, 9, 60)        # e.g. a change score
s = pd.Series(x)
print(s.describe().round(2).to_dict())
print(f"skewness = {stats.skew(x):.2f}  (near 0 -> roughly symmetric, t-tools are fine)")

{'count': 60.0, 'mean': 3.18, 'std': 8.07, 'min': -21.21, '25%': -1.65, '50%': 5.06, '75%': 9.55, 'max': 17.0}
skewness = -0.62  (near 0 -> roughly symmetric, t-tools are fine)

STEP 1 · THE TWO SIDES OF INFERENCE

Estimation and testing answer the same data, two ways

Estimation asks 'what is the value, and how sure are we?' (a point estimate + a confidence interval). Testing asks 'is the effect real?' (H0 vs H1 + a p-value). They are mirror images: if a 95% CI for a difference excludes 0, the matching test rejects H0 at 5%. We let statsmodels' DescrStatsW give the interval, no hand-rolled standard error.

ds = DescrStatsW(x)
lo,hi = ds.tconfint_mean()             # 95% CI from statsmodels
t,p,_ = ds.ttest_mean(0)               # one-sample t-test vs 0
print(f"ESTIMATION  95% CI for the mean: [{lo:.2f}, {hi:.2f}]")
print(f"TESTING     H0: mean=0  ->  t={t:.2f}, p={p:.4f}")
print(f"agree? CI excludes 0 = {lo>0 or hi<0};  test rejects at 5% = {p<0.05}")

ESTIMATION  95% CI for the mean: [1.10, 5.27]
TESTING     H0: mean=0  ->  t=3.05, p=0.0034
agree? CI excludes 0 = True;  test rejects at 5% = True

Always report both: the test says whether there is an effect, the interval says how big it plausibly is. A p-value alone is half an answer. (Throughout this Part we use scipy and statsmodels, the Python equivalents of R and SAS, so the libraries compute the standard errors and test statistics for us.)

STEP 2 · THE MASTER DECISION RULE

Four questions pick the test

Outcome type (numeric vs categorical), number of groups (1 / 2 / 3+), design (independent vs paired), and assumptions (normal-ish vs skewed) determine the test almost uniquely. Here it is as a function you can actually call.

def choose_test(outcome, groups, design="independent", normal=True):
    if outcome=="proportion":
        return "one-proportion z-test" if groups==1 else "two-proportion z-test / chi-square"
    if outcome=="categorical":
        return "chi-square test of independence"
    # numeric outcome
    if groups==1: return "one-sample t-test" if normal else "Wilcoxon signed-rank"
    if groups==2 and design=="paired": return "paired t-test" if normal else "Wilcoxon signed-rank"
    if groups==2: return "two-sample (Welch) t-test" if normal else "Mann-Whitney U"
    return "one-way ANOVA (+Tukey)" if normal else "Kruskal-Wallis"
tests=[("numeric",1,"-",True),("numeric",2,"paired",True),("numeric",2,"independent",False),
       ("numeric",3,"independent",True),("proportion",1,"-",True),("proportion",2,"-",True),("categorical",2,"-",True)]
for o,g,dz,nm in tests: print(f"{o:>11} | {g} groups | {dz:>11} | normal={nm}  ->  {choose_test(o,g,dz,nm)}")

    numeric | 1 groups |           - | normal=True  ->  one-sample t-test
    numeric | 2 groups |      paired | normal=True  ->  paired t-test
    numeric | 2 groups | independent | normal=False  ->  Mann-Whitney U
    numeric | 3 groups | independent | normal=True  ->  one-way ANOVA (+Tukey)
 proportion | 1 groups |           - | normal=True  ->  one-proportion z-test
 proportion | 2 groups |           - | normal=True  ->  two-proportion z-test / chi-square
categorical | 2 groups |           - | normal=True  ->  chi-square test of independence

Memorize the four questions, not the tests; the table falls out of them. The case studies in Chapters 84-88 each run this exact decision out loud before computing anything.

STEP 3 · CHECK THE ASSUMPTIONS FIRST

The fourth question is not optional

Before trusting a t-test or ANOVA, check roughly-normal (or n>=30), and for chi-square check expected counts >= ~5. If they fail, switch to the rank-based twin. Here are quick programmatic checks.

skewed = rng.lognormal(1.0, 0.7, 40)
print(f"skewness = {stats.skew(skewed):.2f}  (|skew|>1 and n<30 -> prefer a rank-based test)")
print(f"Shapiro-Wilk normality p = {stats.shapiro(skewed).pvalue:.4f}  (small p -> not normal)")
tbl = np.array([[8,2],[3,7]])
chi2,p,dof,expected = stats.chi2_contingency(tbl)
print(f"chi-square min expected count = {expected.min():.1f}  ({'OK' if expected.min()>=5 else 'TOO LOW -> use Fisher exact'})")

skewness = 1.41  (|skew|>1 and n<30 -> prefer a rank-based test)
Shapiro-Wilk normality p = 0.0006  (small p -> not normal)
chi-square min expected count = 4.5  (TOO LOW -> use Fisher exact)

A test is only as trustworthy as its assumptions. The honest workflow is check, then choose, never the reverse.

STEP 4 · READ THE RESULT HONESTLY

p-value + effect size + interval, every time

A significant p says an effect is unlikely to be zero; it does NOT say the effect is large or important. Pair every p-value with an effect size and a confidence interval, and remember 'not significant' can just mean underpowered.

big = rng.normal(0.06, 1, 50000)   # a tiny true effect, huge sample
res = stats.ttest_1samp(big, 0); d = big.mean()/big.std()
print(f"p = {res.pvalue:.2e}  (very significant)")
print(f"effect size d = {d:.3f}  (negligible) -> significant but not important")
print("Reporting checklist: estimate, CI, test/p, effect size, assumptions, and the practical decision.")

p = 2.84e-41  (very significant)
effect size d = 0.060  (negligible) -> significant but not important
Reporting checklist: estimate, CI, test/p, effect size, assumptions, and the practical decision.

That is the whole of inference in one habit: choose the right test for the data, then report the size of the effect with its uncertainty, not just whether p cleared 0.05.

📋 STATISTICIAN’S REPORT

Your roadmap through the case studies

Chapters 84-88 each take ONE real dataset from question to plain-English recommendation, narrating every decision. Ch84 a web A/B test (two-proportion z, power, lift CI). Ch85 comparing marketing channels (one-way ANOVA + Tukey). Ch86 a clinical trial (paired, two-arm, and responder-rate tests on one trial). Ch87 a customer survey (chi-square, a proportion with a margin of error, and an ordinal test). Ch88 manufacturing quality (one-sample vs spec, ANOVA across lines, defect-by-shift chi-square). Watch how the same four questions choose the tool every time.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 83 · Review & Choosing the Right Test