←Back to the bookChapter 66 · Study Design & Data Quality

Chapter 66 · Solutions

Study Design & Data Quality — Worked Solutions ✅

Five challenges, each verified in code.

⚙️ Setup

import numpy as np, pandas as pd
rng = np.random.default_rng(650)

CHALLENGE 1

Spurious correlation from a confounder

Let a confounder Z drive both X and Y. Show X and Y correlate, then control for Z and watch it vanish.

Z = rng.uniform(0,10,2000)
X = 2*Z + rng.normal(0,2,2000)
Y = 3*Z + rng.normal(0,3,2000)
def resid(v): return v - np.poly1d(np.polyfit(Z,v,1))(Z)
print(f"raw corr(X,Y)            = {np.corrcoef(X,Y)[0,1]:.2f}")
print(f"corr after removing Z    = {np.corrcoef(resid(X),resid(Y))[0,1]:.2f}")

raw corr(X,Y)            = 0.90
corr after removing Z    = 0.05

CHALLENGE 2

Randomization balances a confounder

Compare the treated/control means of a covariate under self-selection vs a coin-flip assignment.

n=5000; age=rng.uniform(20,70,n)
self_sel = rng.random(n) < 1/(1+np.exp(-(age-45)/8))
rand     = rng.random(n) < 0.5
print(f"self-selected: treated age {age[self_sel].mean():.0f} vs control {age[~self_sel].mean():.0f}  (imbalanced)")
print(f"randomized   : treated age {age[rand].mean():.0f} vs control {age[~rand].mean():.0f}  (balanced)")

self-selected: treated age 54 vs control 36  (imbalanced)
randomized   : treated age 45 vs control 45  (balanced)

CHALLENGE 3

Build your own Simpson's paradox

Construct two groups where Treatment X beats Y in each subgroup but loses overall.

# X: great on easy cases but mostly given hard cases; Y: the reverse
rows = pd.DataFrame({"trt":["X","X","Y","Y"],"grp":["easy","hard","easy","hard"],
                     "succ":[19,41,85,5],"tot":[20,80,100,10]})
rows["rate"]=rows.succ/rows.tot
print(rows.to_string(index=False))
ov = rows.groupby("trt").apply(lambda g:g.succ.sum()/g.tot.sum(), include_groups=False)
print(f"\neasy: X {19/20:.0%} vs Y {85/100:.0%}; hard: X {41/80:.1%} vs Y {5/10:.0%}")
print(f"overall: X {ov['X']:.0%} vs Y {ov['Y']:.0%}  -> reversal!")

trt  grp  succ  tot   rate
  X easy    19   20 0.9500
  X hard    41   80 0.5125
  Y easy    85  100 0.8500
  Y hard     5   10 0.5000

easy: X 95% vs Y 85%; hard: X 51.2% vs Y 50%
overall: X 60% vs Y 82%  -> reversal!

CHALLENGE 4

Data quality audit

Audit a messy dataframe for completeness, uniqueness, and validity.

df = pd.DataFrame({"id":[1,2,2,4],"score":[88, np.nan, 75, 150]})
print(f"completeness (score)   : {1-df.score.isna().mean():.0%}")
print(f"uniqueness (ids)       : {df.id.nunique()/len(df):.0%}")
print(f"validity (score 0-100) : {df.score.between(0,100).mean():.0%}")

completeness (score)   : 75%
uniqueness (ids)       : 75%
validity (score 0-100) : 50%

CHALLENGE 5

Bias is a design flaw, not a sample-size one

Show a confounded estimate stays biased as n grows from 2,000 to 50,000.

for n in [2000, 50000]:
    age=rng.uniform(20,70,n); base=100-0.5*age
    tr = rng.random(n) < 1/(1+np.exp(-(age-45)/8))
    out = base + 8.0*tr + rng.normal(0,5,n)
    print(f"n={n:>6}: naive estimate {out[tr].mean()-out[~tr].mean():+.1f}  (true 8.0)")
print("bias constant: more data cannot fix a flawed design")

n=  2000: naive estimate -1.0  (true 8.0)
n= 50000: naive estimate -1.1  (true 8.0)
bias constant: more data cannot fix a flawed design

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 66 · Study Design & Data Quality