"52% support, ± 3 points." That little ± is the margin of error, and it is the most widely seen, and most widely misunderstood, number in statistics. This chapter is a close look at exactly what it is and what makes it big or small.
Every interval in this Part is estimate ± margin. The margin of error is that ±. Understanding it is understanding how precise an estimate really is, and how to report it honestly.
What the Margin of Error Is
A confidence interval is estimate ± margin. That margin is the margin of error, E = z·SE. When a poll reports "52% ± 3%", the 3 points is E, almost always at 95% confidence.
The margin of error tells you how precise the estimate is, nothing more and nothing less. It does not describe error from bad questions, biased samples, or non-response, only the random sampling uncertainty. A small margin on a biased sample is still wrong (recall Bias in Data Collection); E measures precision, not accuracy.
The Three Drivers
For a proportion, E = z·√(p(1−p)/n). Three things move it.
| Driver | Effect on E | From the notebook (n = 1,000, p = 0.5) |
|---|---|---|
| Confidence level (z) | higher confidence → wider E | 90% ±2.6 · 95% ±3.1 · 99% ±4.1 pts |
| Variability p(1−p) | most at p = 0.5 | p=0.1 ±1.9 · p=0.3 ±2.8 · p=0.5 ±3.1 pts |
| Sample size n | more data → smaller E (1/√n) | only knob that helps without costing confidence |
Two of the three are trade-offs: you cannot lower the margin by lowering your confidence (that is cheating), and you do not control how variable the population is, so planners assume the worst case p = 0.5, where p(1−p) is largest. The only honest lever is sample size.
The 1/√n Law
Because E is proportional to 1/√n, precision is expensive: each halving of the margin costs four times the sample.
Going from ±6 to ±3 points is cheap; going from ±3 to ±1.5 costs four times as much again. This is exactly why the classic national poll lands near n = 1,000 (about ±3%): it is the sweet spot where extra precision stops being worth the cost. The same curve governs how big a test set you need to certify a model's accuracy.
Real-World Example: Reporting a Survey
A company surveyed 900 customers and wants to announce the share who would recommend it, the kind of number that goes in a press release or board deck. Reporting it with a margin of error is what makes it credible.
One row per respondent with a rating_1_5 and a derived
would_recommend (top-2-box).
The result, reported the right way:
"64% of customers would recommend us, ± 3.1 points (95% confidence)." The true rate is very likely between 61.2% and 67.5%. Reporting only "64%" would imply a precision the data does not have.
| Lever | Setting | Margin of error |
|---|---|---|
| Confidence | 90% / 95% / 99% | ±2.6 / ±3.1 / ±4.1 pts |
| Sample size | n = 300 / 900 / 2,500 | ±5.4 / ±3.1 / ±1.9 pts |
The levers are clear: demanding 99% confidence widens the margin to about ±4 points, and to roughly halve the margin the company would need to survey nearly three times as many customers. Knowing these trade-offs is the difference between quoting a number and defending it.
Margin of Error in Machine Learning & AI
The same ± belongs on every metric a model reports. A leaderboard number without a margin is a poll without one.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Margin of error | Error bars on a metric | accuracy 92% ± 1.5% |
| 1/√n law | How big a test set you need | ±1% accuracy needs ~9,600 labeled items |
| Worst case p = 0.5 | Conservative eval sizing | plan test-set size assuming 50% accuracy |
| Precision vs confidence | How tight a claim you can make | tighter bars require more eval data |
A model's accuracy is a proportion, so it has a margin of error exactly like a poll. With a 400-item test set, the 95% margin is roughly ±5 points, so "91%" and "94%" models are indistinguishable. The 1/√n law tells you the test set must quadruple to halve those error bars, which is why certifying a 1% improvement requires on the order of 10,000 labeled examples. Reporting a metric without its margin of error is the leaderboard equivalent of a poll with no ±.
Dissect the margin of error in Python
The companion notebook computes E = z·SE, varies all three drivers (confidence, variability, n), plots
the 1/√n curve, and loads margin-of-error--customer_survey.xlsx to report a real recommend rate with
its margin of error and a levers table.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Margin of error E = z·SE is the ± half-width of a confidence interval, the precision of an estimate.
- ✓Three drivers: confidence (bigger z → wider), variability (worst at p=0.5), and sample size.
- ✓1/√n law: quadruple the sample to halve the margin; the classic poll (n≈1,000) is ±3%.
- ✓Precision ≠ accuracy: a small margin on a biased sample is still wrong; E ignores question and sampling bias.
- ✓Real survey: 64% would recommend ±3.1 points at 95%; in ML, a 400-item test set gives ±5 points, so report metric margins.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and SciPy before checking the solutions.
Compute a margin of error
A poll of 1,200 finds 47% support. Compute the 95% margin of error.
Worst-case margin
Not knowing p, compute the worst-case 95% margin for n = 1,200 (use p = 0.5).
Confidence widens the margin
For n = 1,000 and p = 0.5, tabulate the margin at 90%, 95%, and 99% confidence.
stats.norm.ppf(0.5 + conf/2).Quadruple for half
Confirm that going from n = 1,000 to n = 4,000 halves the margin of error.
Real data: survey margin
Load margin-of-error--customer_survey.xlsx and report the recommend rate with its 95% margin of error.
would_recommend, then z·SE.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on the margin of error. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Every interval so far relied on a formula. The Resampling & Simulation chapter closes the Part with the bootstrap, which builds a confidence interval for almost any statistic, including ones with no formula at all, by letting the data resample itself.