Sampling decides who is in your data; study design decides what the data can prove. A perfectly sampled, perfectly clean dataset can still lead you to a false conclusion if the study was designed to only observe. This chapter is about turning data into trustworthy answers.
Correlation is not causation, randomization is what bridges the gap, a lurking variable can even reverse a conclusion (Simpson's paradox), and none of it matters if the data itself is dirty. Design and quality come before analysis.
Observational vs Experimental Studies
The deepest divide in data collection is whether you observe or intervene. Observation is cheap and often the only option, but it is vulnerable to confounding: a lurking variable that drives both things you are comparing.
The notebook makes the classic case concrete: ice-cream sales and drownings correlate at 0.58, but once temperature is held constant the correlation collapses to −0.03. Neither causes the other; hot weather causes both. This is why an observational association, however strong, can never by itself prove causation.
Randomization & Control
An experiment assigns the treatment. The magic ingredient is random assignment: deciding each subject's group by a coin flip makes the treatment and control groups statistically identical on every variable at once, so any difference in outcome must be the treatment's doing.
In the notebook, when older subjects self-select into treatment the naive comparison gives −1.5, badly wrong, because the treated group is 19 years older. Flip a coin instead and the groups match on age (45 vs 45), so the estimate lands on the true effect of +8.0. A randomized controlled trial with a control group and, ideally, blinding, is the gold standard for establishing cause.
Confounding & Simpson's Paradox
Confounding has a spectacular extreme: an association can run one way overall and the opposite way in every subgroup. The real kidney-stone data of Charig et al. (1986) is the textbook case.
Treatment A succeeds more often for small stones (93% vs 87%) and for large stones (73% vs 69%), yet B beats A overall (83% vs 78%). The cause is the confounder stone size: doctors gave the tougher open surgery (A) to far more of the hard large-stone cases. Aggregating across a confounder can flip the truth, so before trusting any overall comparison, ask what variable might be hiding inside it.
Data Quality Dimensions
Even a flawless design fails on dirty data. Quality is not one thing but several, and a dataset is only as trustworthy as its weakest dimension.
| Dimension | Question it answers | Example failure (from the audit) |
|---|---|---|
| Completeness | Are values present? | a missing age (83% complete) |
| Uniqueness | Are records de-duplicated? | a duplicated id (83% unique) |
| Validity | Are values in the legal range? | an age of 200 (67% valid) |
| Consistency | Same units & format throughout? | a height of 1.75 (meters, not cm) |
| Timeliness / format | Parseable and up to date? | an unreadable "bad-date" |
| Accuracy | Do values match reality? | a typo'd measurement (hard to detect) |
The notebook audits a small dataset and scores each dimension separately, catching the missing value, the duplicate row, the impossible age, the wrong-unit height, and the malformed date. Crucially, these are different failures needing different checks: a single "percent clean" number would hide them. Profiling every dimension is the first step of any serious analysis.
Study Design & Data Quality in Machine Learning & AI
Machine learning is observational by default: models learn from whatever data they are fed, confounders and all. The hardest failures in applied ML are design and data-quality failures, not modeling failures.
| Idea (this chapter) | In ML / AI it appears as | The danger |
|---|---|---|
| Confounding | Spurious features / shortcut learning | the model keys on a confounder, not the cause |
| Randomized experiment | A/B tests & randomized rollouts | the only way to measure true causal lift |
| Simpson's paradox | Aggregate vs per-segment metrics | overall accuracy hides per-group failures |
| Data quality | Validation, profiling, leakage checks | garbage in, garbage out, at scale |
| Design > sample size | Better data beats more data | more biased data does not help |
The notebook's blunt finding, a confounded estimate is just as wrong with 40,000 rows as with 4,000, is the deepest lesson in applied ML: garbage in, garbage out, and scaling the garbage does not help. Models exploit confounders as shortcuts (the famous case of a classifier that detected the ruler in skin-cancer photos rather than the lesion). Establishing that a feature or intervention truly causes an outcome requires a randomized A/B test, not an observational correlation. Simpson's paradox warns that headline accuracy can mask severe per-segment failures, the heart of fairness auditing. And data leakage, a validity failure where future or target information sneaks into the features, is the most common cause of models that look brilliant in development and collapse in production. Design and data quality are where real ML projects are won or lost.
See design and quality in Python
The companion notebook manufactures a spurious correlation and controls it away, contrasts a self-selected estimate with a randomized one, reproduces Simpson's paradox from the real kidney-stone data, audits a dataset across the quality dimensions, and shows confounding bias surviving a tenfold increase in data.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, and
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Observation shows association; experiments show causation, because a confounder can fake a correlation (r 0.58 → −0.03 once controlled).
- ✓Randomization balances all confounders: a self-selected estimate was −1.5, the randomized one +8.0 (the truth).
- ✓Simpson's paradox: A beat B in both subgroups (93/73 vs 87/69) yet lost overall (78% vs 83%), the confounder was stone size.
- ✓Data quality is multi-dimensional: completeness, uniqueness, validity, consistency, timeliness, accuracy, each needs its own check.
- ✓Design beats cleanup: a confounded estimate stayed wrong with 10× the data; in ML, garbage in is garbage out at scale.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and pandas before checking the solutions.
Spurious correlation from a confounder
Let a confounder Z drive both X and Y. Show X and Y correlate, then control for Z and watch it vanish.
Randomization balances a confounder
Compare the treated/control means of a covariate under self-selection versus a coin-flip assignment.
Build your own Simpson's paradox
Construct two treatments where X beats Y in each subgroup but loses overall.
Data quality audit
Audit a messy dataframe for completeness, uniqueness, and validity, and report each as a percentage.
isna, nunique, and between.Bias is a design flaw, not a sample-size one
Show a confounded estimate stays biased as n grows from 2,000 to 50,000.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on study design and data quality. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Good design assumes honest measurement, but collection itself can distort the data. Bias in Data Collection closes the Part: selection, nonresponse, response, and the survivorship bias of Wald's wartime airplanes.