Every analysis so far assumed you had data. This Part asks the prior question: where does trustworthy data come from? It starts with a deceptively simple idea, sampling, studying a subset to learn about the whole.
Probability through Probability & Distributions Case Studies gave you the mathematics of probability and distributions. Now we turn practical: how to gather data worth analyzing. This chapter makes the case for sampling; the next ones cover how to sample well, how much to collect, and the biases that ruin data.
The Census Problem
A census measures every unit in the population. It sounds ideal, and occasionally it is worth it (a national population count). But for most questions it is impractical, and sometimes outright impossible.
| Why a census fails | Example |
|---|---|
| Cost | interviewing all 330 million residents of a country |
| Time | the answer would be stale before the count finished |
| Destructive testing | you cannot crash-test every car or taste-test every batch |
| Inaccessibility | the population is infinite, hypothetical, or future (all possible users) |
Sampling sidesteps all four. The catch is that a sample introduces uncertainty, and the rest of this chapter shows that the uncertainty is small, measurable, and well worth the savings.
A Sample Stands In for the Population
The magic of a representative sample is that it mirrors the population in miniature, so a statistic computed on the sample estimates the matching population parameter.
In the notebook a population of 1,000,000 incomes has a true mean of about $58,700. A single random sample of just 1,000 people, one-thousandth of the population, estimates it as $58,000, off by only about 1%. That is the promise: a sliver of the data carries almost all the signal, provided the sample is drawn fairly so it represents everyone.
Sampling Error & the Square-Root Payoff
No sample is perfect. Sampling error is the random gap between a sample statistic and the true parameter. For the mean, its typical size is the standard error, SE = σ/√n, and it shrinks in a very specific way as the sample grows.
The notebook confirms it precisely: the observed spread of sample means tracks σ/√n almost exactly (at n = 2,000, the observed SE is about $849 against a predicted $862). Crucially, the standard error depends on n, not on the fraction n/N. Sampling 1,000 people gives the same precision whether the population is 10,000 or 1,000,000:
| Population N | Fraction sampled (n = 1,000) | Standard error |
|---|---|---|
| 10,000 | 10% | ~$1,166 |
| 100,000 | 1% | ~$1,188 |
| 1,000,000 | 0.1% | ~$1,208 |
This is why a national poll of a few thousand can speak for an entire country. Precision rides on the absolute sample size, and the cost is quadratic, so returns diminish quickly.
Representativeness Beats Size
If a sample is drawn unfairly, no amount of data saves it. The classic cautionary tale is the 1936 Literary Digest poll: it mailed 10 million ballots and tallied 2.4 million, predicting Landon would crush Roosevelt. Roosevelt won in a landslide. George Gallup, polling just 50,000 people representatively, called it correctly.
The notebook reproduces the effect: with true support at 55%, a biased frame of 240,000 people estimates 46% (wrong), while a simple random sample of just 2,000 gets 54% (right). The biased sample's error is systematic, so 120 times more data only sharpens a wrong answer. And because precision shows diminishing returns, doubling a sample from 2,000 to 4,000 raises precision by only about 41%. The lesson that drives the rest of this Part: how you sample matters far more than how much.
Sampling in Machine Learning & AI
Sampling is not just for surveys. Modern machine learning is built on it, every model is trained on a sample and evaluated on another, and the same lessons, error shrinks like 1/√n and bias cannot be outvoted, apply directly.
| Idea (this chapter) | In ML / AI it becomes | Concrete example |
|---|---|---|
| Sample estimates population | Train / test split | a held-out test set estimates real-world accuracy |
| Random subsampling | Mini-batch gradient descent | each batch is a random sample; gradient noise ∝ 1/√batch |
| Resampling | The bootstrap | resample the data to estimate uncertainty and build ensembles |
| Representativeness beats size | Data quality over data quantity | a biased web scrape misleads no matter how big |
| Smart sampling | Active learning | label the most informative points, not the most |
A test set is a sample used to estimate a model's true performance, so its accuracy has a standard error of its own, report it, and a test set of 100 examples cannot certify a 1% improvement. Mini-batch training is sampling in a loop: gradient noise falls like 1/√(batch size), the same square-root law. Most importantly, the Literary Digest lesson is the modern crisis of biased training data: a model learns whatever sample it is fed, and a huge but unrepresentative dataset bakes in a confident, systematic error. In AI as in surveys, a smaller representative dataset beats a massive skewed one.
See sampling work in Python
The companion notebook builds a million-person population, estimates its mean from a sample of 1,000, traces the standard error down the square-root law, proves the sample fraction barely matters, recreates the Literary Digest bias, and plots the diminishing returns of ever-larger samples.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy and matplotlib and launch
jupyter notebook.
🎓 Key Takeaways
- ✓A census (measuring everyone) is often too costly, slow, or impossible; a sample estimates the whole from a part.
- ✓A statistic estimates a parameter: x̄ for μ, p̂ for p; a sample of 1,000 pinned a million-person mean to within ~1%.
- ✓Standard error = σ/√n: quadruple the sample to halve the error, and it depends on n, not the fraction sampled.
- ✓Representativeness beats size: a biased frame is wrong no matter how large, because its error is systematic.
- ✓In ML/AI: test sets, mini-batches, and the bootstrap are all sampling; biased training data is the Literary Digest of our era.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy before checking the solutions.
The sampling-error law
Draw 2,000 samples of size 50 from Normal(100, 20) and check the observed standard error against σ/√n.
std(), compare to 20/√50.The fraction myth
Draw n = 200 from a population of 2,000 and from one of 200,000. Show the standard error is essentially the same.
Bias cannot be outvoted
A convenience method reaches only group A (mean 60) when the population is half A, half B (mean 40). Show the estimate stays wrong as n grows.
Quadruple for half
Show the standard error at n = 400 is about half the standard error at n = 100.
Precision per dollar
If measuring one unit costs $1, compare the precision (1/SE) of a census of 1,000,000 to a sample of 2,000.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on why we sample. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
We have made the case for sampling and seen that representativeness is everything. The next chapter, Probability Sampling Methods, shows the probability sampling methods, simple random, stratified, cluster, and systematic, that actually deliver a representative sample.