←Back to the bookChapter 66 · Bias in Data Collection

Chapter 66 · Solutions

Bias in Data Collection — Worked Solutions ✅

Five challenges, each verified in code.

⚙️ Setup

import numpy as np, pandas as pd
rng = np.random.default_rng(660)
N=200_000

CHALLENGE 1

Coverage bias

Build a frame that omits younger people and show it mis-estimates an age-linked trait.

age = rng.integers(18,85,N)
trait = rng.random(N) < (0.7-0.005*(age-18))
in_frame = rng.random(N) < (age-15)/80     # older more likely in frame
print(f"true trait    = {trait.mean():.1%}")
print(f"frame estimate= {trait[in_frame].mean():.1%}  (bias {(trait[in_frame].mean()-trait.mean())*100:+.1f} pts)")

true trait    = 53.4%
frame estimate= 48.1%  (bias -5.3 pts)

CHALLENGE 2

Nonresponse bias and the weighting fix

High earners respond less. Show the observed mean is biased low, then repair it with inverse-propensity weights.

income = rng.lognormal(10.7,0.6,N)
resp_p = 0.8 - 0.5*(income > np.quantile(income,0.7))
r = rng.random(N) < resp_p
naive = income[r].mean()
weighted = np.average(income[r], weights=1/resp_p[r])
print(f"true     ${income.mean():,.0f}")
print(f"observed ${naive:,.0f}  (bias {(naive/income.mean()-1)*100:+.1f}%)")
print(f"weighted ${weighted:,.0f}  (bias {(weighted/income.mean()-1)*100:+.1f}%)")

true     $52,981
observed $43,578  (bias -17.7%)
weighted $52,935  (bias -0.1%)

CHALLENGE 3

Response (social-desirability) bias

Model self-reports that inflate a true behavior and quantify the gap.

true = np.clip(rng.normal(3,2,N),0,None)
reported = true + np.clip(rng.normal(2,1,N)-0.3*true,0,None)
print(f"true mean     {true.mean():.2f}")
print(f"reported mean {reported.mean():.2f}  (response bias {reported.mean()-true.mean():+.2f})")

true mean     3.06
reported mean 4.25  (response bias +1.19)

CHALLENGE 4

Survivorship bias

Hits land uniformly across plane regions, but engine hits are usually fatal. Show returning planes look least-hit on the engine.

planes=6000; regions=["engine","wing","tail"]; fatal={"engine":0.7,"wing":0.1,"tail":0.1}
hit = rng.integers(0,3,planes)
survived = np.array([rng.random()>fatal[regions[h]] for h in hit])
allc = pd.Series([regions[h] for h in hit]).value_counts().reindex(regions)
surv = pd.Series([regions[h] for h in hit[survived]]).value_counts().reindex(regions)
print("all planes (true):\n", allc.to_string())
print("\nreturning planes (observed):\n", surv.to_string())
print("\nengine looks safest among survivors -> that is exactly where to armor")

all planes (true):
 engine    1962
wing      2036
tail      2002

returning planes (observed):
 engine     587
wing      1814
tail      1792

engine looks safest among survivors -> that is exactly where to armor

CHALLENGE 5

The feedback loop

Show a model trained on biased historical hiring reproduces the unfair gap.

n=20000; group=rng.integers(0,2,n); skill=rng.normal(50,10,n)
hist = ((group==0)&(skill>50)) | ((group==1)&(skill>62))
print(f"historical: group0 {hist[group==0].mean():.0%}, group1 {hist[group==1].mean():.0%}")
t0=np.quantile(skill[group==0],1-hist[group==0].mean()); t1=np.quantile(skill[group==1],1-hist[group==1].mean())
model=((group==0)&(skill>t0))|((group==1)&(skill>t1))
print(f"model:      group0 {model[group==0].mean():.0%}, group1 {model[group==1].mean():.0%}  (bias reproduced)")

historical: group0 50%, group1 11%
model:      group0 50%, group1 11%  (bias reproduced)

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 66 · Bias in Data Collection