"How many people do we need?" is the first question every survey, experiment, and study must answer. Too few and the result is uselessly vague; too many and you waste money. The answer is not a guess, it follows from the precision you require.
Every sample-size calculation needs the same three things: a confidence level (fixes z), a target margin of error E, and a measure of variability (σ for a mean, p for a proportion). Supply those and n is determined.
The Margin of Error
A confidence interval is estimate ± margin of error, where E = z·SE. The margin is the precision knob, and you set its value before collecting a thing.
With σ = 15, the notebook shows the margin tightening from ±4.16 at n = 50 to ±1.47 at n = 400. To hit a target margin, you simply invert E = z·σ/√n and solve for n, which is the whole game.
Sample Size for a Mean
Solving the margin formula for n gives the sample-size rule for a mean:
For σ = 15 and a target margin of ±2 at 95% confidence, the formula gives n = 217, and the notebook's simulation confirms that samples of that size really do produce intervals of half-width ±2.0. The one wrinkle: you need an estimate of σ in advance, from a pilot study, prior research, or a conservative guess.
Sample Size for a Proportion
For a yes/no question the variability is p(1−p), which is largest at p = 0.5. Since you rarely know p before sampling, planners use 0.5 for the safest, largest, sample size.
The worst-case sample size, with p = 0.5 and a ±3% margin at 95%, is n ≈ 1,067, which is exactly why the textbook national poll surveys "about a thousand" people. If a pilot tells you p is far from 0.5, you need fewer: at p = 0.1 the same margin needs only 384. Assuming 0.5 simply guarantees the margin whatever the truth turns out to be.
Diminishing Returns & the Finite-Population Correction
Because n grows as 1/E², precision is expensive: each halving of the margin demands four times the data.
Tightening the margin from ±2% to ±1% multiplies the required sample by four, from 2,401 to 9,604. The finite-population correction, n = n₀/(1 + (n₀−1)/N), gives some back when the sample is a sizable fraction of a small population: for a town of 2,000 the needed sample drops from 1,067 to 696, but for a population of 100,000 it barely changes. Big populations get no discount.
Power, A/B Tests & Machine Learning
Margin of error sizes a single estimate. To compare groups, an A/B test, a treatment effect, you size for statistical power: the probability of detecting a real effect when it exists. This is the sample-size question all over machine learning.
| Idea (this chapter) | In ML / AI it becomes | Concrete example |
|---|---|---|
| Margin of error | Confidence on a metric | a test set of n gives accuracy ± z√(acc(1−acc)/n) |
| Sample size for a proportion | How big a test set? | to certify ±1% accuracy needs ~9,600 labeled examples |
| Statistical power | A/B test & experiment sizing | ~4,000/group to detect a 2-point lift at 80% power |
| 1/E² cost law | Diminishing returns of labels | each extra digit of precision costs 100× the data |
A model's reported accuracy is an estimate with a margin of error, so a test set has a required size just like a poll: certifying accuracy to ±1% needs on the order of 9,600 labeled examples, and a 100-item test set cannot tell a 90% model from a 93% one. Comparing two models or two prompts is an A/B test, and the notebook shows detecting a 2-point lift takes roughly 4,000 per group for 80% power, which is why so many "our model is better" claims are statistically underpowered. Power analysis before you run the experiment is what separates a real result from noise.
Compute (and verify) sample sizes in Python
The companion notebook derives n for a mean and a proportion, confirms by simulation that the resulting intervals hit the target margin, plots the p(1−p) parabola and the 1/E² cost curve, applies the finite-population correction, and runs a power analysis for an A/B test.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, scipy, and
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Margin of error E = z·SE is the precision you choose; sample size comes from inverting it.
- ✓For a mean: n = (z·σ/E)² (here 217 for σ=15, ±2); you need an estimate of σ first.
- ✓For a proportion: n = z²p(1−p)/E², worst case p=0.5 gives ~1,067 for ±3%, the classic poll size.
- ✓Cost is 1/E²: halving the margin quadruples n; the finite-population correction helps only for small populations.
- ✓Power sizes comparisons (A/B tests, model evals): detecting a 2-point lift needs ~4,000/group at 80% power.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and SciPy before checking the solutions.
Sample size for a mean
With σ = 20 and a target margin of ±3 at 95% confidence, find n and verify it by simulation.
Sample size for a proportion
A pilot suggests p = 0.2. For a ±4% margin at 95%, find n.
The worst case is p = 0.5
Show that p = 0.5 maximizes the required sample size, and compute the safe n for ±4%.
Halve the margin, quadruple the sample
Confirm that cutting the target margin in half multiplies the required sample size by four.
Higher confidence costs more
Compare the sample size for 95% vs 99% confidence at the same ±3% margin.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on sample size. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now size a study. the Study Design & Data Quality chapter steps back to study design and data quality: observational versus experimental studies, randomization, confounding, and the dimensions that make data trustworthy in the first place.