We close the Part with the failure mode that haunts every dataset: bias. It is not a mistake in arithmetic or a small sample, it is a systematic tilt introduced by the collection process itself, and it survives any amount of extra data.
This chapter's centerpiece is Abraham Wald's WWII analysis of bullet holes on returning aircraft, the most famous illustration that the data you can see is not the data you need to see. Reasoning about the missing data is the whole skill.
Selection & Coverage Bias
Coverage bias arises when the sampling frame, the list you actually draw from, does not cover the whole population. Anyone outside the frame has zero chance of selection, so their values never enter, no matter how carefully you sample.
In the notebook a landline-style frame has a mean age of 61 against the population's 51, and since support for the policy falls with age, the frame under-estimates support by 6.2 points (49% vs the true 55%). Undercoverage is invisible inside the data, you can only catch it by asking who is missing from the list.
Nonresponse Bias
Even with a perfect frame, the people who decline can differ systematically from those who answer. Nonresponse bias is the gap between responders and the population, and it grows as the response rate falls.
When high earners refuse, the observed mean income falls to $43,700, a −17.7% bias against the true $53,200. If the response pattern is known, inverse-propensity weighting up-weights the under-represented responders and restores the estimate to $53,200. When the pattern is unknown, as it usually is, the bias simply remains, which is why survey teams fight so hard for a high response rate.
Response & Measurement Bias
Sometimes the sample is fine but the answers are skewed. Response bias and measurement bias come from the instrument: leading questions, social pressure to give a flattering answer, or faulty recall.
| Source | What it does | Remedy |
|---|---|---|
| Leading / loaded wording | steers respondents toward an answer | neutral, balanced question wording |
| Social desirability | inflates "good" behavior, hides "bad" | anonymity, indirect questioning |
| Recall bias | memory distorts past events | shorter recall windows, records |
| Acquiescence | tendency to agree with any statement | reverse-worded items, validated scales |
The notebook models social desirability on a survey of weekly exercise: the true mean is 3.06 hours, but self-reports average 4.25 hours, an inflation of +1.19 hours. No sampling fix touches this, because every respondent gives a skewed answer. The cure is a better instrument: neutral wording, anonymity, and validated scales.
Survivorship Bias: Wald's Airplanes
The most elegant bias of all. In World War II, the US military studied bullet holes on bombers returning from raids and proposed adding armor where the holes clustered. The statistician Abraham Wald saw the fatal flaw: those were the planes that survived. The data was conditioned on coming back.
In the notebook, the engine takes its fair share of hits across all planes (1,036), but among returning planes it shows the fewest holes (301), because engine hits brought planes down. The naive reading, "armor the wings and fuselage where the holes are", is exactly backwards. Wald's conclusion: armor the engine and cockpit, the places survivors are unscathed, because planes hit there did not survive. The sample is conditioned on survival, and the absent cases hold the answer.
Bias in Machine Learning & AI
Every bias in this chapter reappears in machine learning, magnified by scale and automation. A model is a mirror of its training data, so collection bias becomes model bias, and a deployed model can feed its bias back into the next dataset.
| Bias (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Coverage bias | Under-represented groups in training data | face systems weak on under-sampled skin tones |
| Nonresponse bias | Missing-not-at-random labels | only satisfied users leave ratings |
| Survivorship bias | Training only on outcomes that "made it" | funded startups, completed loans, clicked ads |
| Feedback loops | Model outputs become future training data | biased hiring/lending perpetuated at scale |
Wald's airplanes are a parable for modern ML: models are trained on the cases that survived into the dataset, funded startups, approved loans, users who stayed, and silently miss the ones that did not, the archetype of survivorship bias. In the notebook, a hiring model trained on biased history reproduces the unfair gap exactly (group 0 hired 50%, group 1 just 11%), and once deployed it generates the next round of biased records, the feedback loop that entrenches discrimination. No amount of additional data fixes this, because the bias is in what was collected. The remedies are deliberate: audit performance by subgroup, reweight or rebalance the data, debias labels, add fairness constraints, and above all reason about the cases that never made it into the dataset. Representative, well-measured collection is the foundation of trustworthy AI.
Quantify each bias in Python
The companion notebook measures coverage bias from a skewed frame, nonresponse bias and its weighting fix, social-desirability response bias, recreates Wald's survivorship-bias airplanes, and shows a model reproducing historical hiring bias, the feedback loop in action.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, and
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Bias is systematic error from how data is collected; it does not shrink with more data, more data makes you more confidently wrong.
- ✓Coverage bias: the frame misses part of the population (a landline frame under-counted support by 6.2 points).
- ✓Nonresponse bias: refusers differ (income off −17.7%); inverse-propensity weighting can fix a known pattern. Response bias skews the answers themselves (+1.19 h exercise).
- ✓Survivorship bias: Wald's planes, the engine looked safe only because hit planes never returned; armor where survivors are unscathed.
- ✓In ML/AI: collection bias becomes model bias and feeds a loop (hiring gap 50% vs 11% reproduced); reason about the data you cannot see.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and pandas before checking the solutions.
Coverage bias
Build a frame that omits younger people and show it mis-estimates an age-linked trait.
Nonresponse bias and the weighting fix
Let high earners respond less. Show the observed mean is biased low, then repair it with inverse-propensity weights.
Response (social-desirability) bias
Model self-reports that inflate a true behavior and quantify the gap.
Survivorship bias
Hits land uniformly across plane regions, but engine hits are usually fatal. Show returning planes look least-hit on the engine.
The feedback loop
Show a model trained on biased historical hiring reproduces the unfair group gap.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on bias in data collection. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
We have seen the biases that corrupt data as it is gathered, but a survey is only one way to "get" data. The Modes of Data Collection chapter steps back to the full spectrum of data-collection modes, sensors, web scraping, direct observation, digital-trace logs, and administrative records, each with its own collection design and a worked Excel example.