A great deal of real decision-making rests on a yes/no rate, did the visitor convert, the patient recover, the voter approve? This chapter builds confidence intervals for those proportions, and, more usefully, for the difference between two of them.
Two separate intervals are not the test. The right tool is a single interval on the difference, and the decision rule is simple: if that interval excludes zero, the difference is real.
Confidence Interval for a Proportion
For a yes/no outcome, the sample proportion p̂ = successes/n estimates the true rate p. Its standard error is √(p̂(1−p̂)/n), so the 95% interval is p̂ ± 1.96·SE.
The success-failure condition, at least about 10 successes and 10 failures, is what justifies the normal approximation. In the notebook, 84 of 500 gives 16.8% with a 95% interval of [13.5%, 20.1%]. With very few successes the approximation breaks down, and the Wilson or exact (Clopper-Pearson) interval is safer.
Difference of Two Proportions
The interesting quantity is usually the lift, pB − pA. Its standard error adds the two groups' variances: SE = √(pA(1−pA)/nA + pB(1−pB)/nB). The decision turns on whether the interval excludes 0.
If the whole interval sits above 0, group B's rate is genuinely higher; if it straddles 0, you cannot rule out "no difference". This single rule, does the difference interval exclude 0?, is the backbone of A/B testing and is exactly equivalent to the hypothesis tests of the next Part.
Difference of Two Means
When the outcome is numeric, revenue per user, minutes on site, the same idea uses a two-sample t-interval: (x̄B − x̄A) ± t*·√(sA²/nA + sB²/nB), with degrees of freedom from the Welch approximation.
In the notebook, two spending groups differ by +$5.09 with a 95% interval of about [+$2.80, +$7.38], again clear of zero. Whether the target is a rate or an average, the interval on the difference answers the real question: is the gap big enough to be sure it is not noise?
Real-World Example: An A/B Test
A product team ran a randomized experiment and exported 2,000 visitors: variant A is the current page, variant B a new design. Did B actually convert better, or did it just look better by chance?
One row per visitor with variant (A/B) and converted (0/1).
| Variant | n | Conversions | Rate | 95% CI |
|---|---|---|---|---|
| A (control) | 995 | 109 | 10.95% | [9.01%, 12.90%] |
| B (new design) | 1,005 | 147 | 14.63% | [12.44%, 16.81%] |
| Lift (B − A) | — | — | +3.67 pts | [+0.75, +6.59] pts |
The lift is +3.7 percentage points, and its 95% interval, roughly [+0.8, +6.6] points, lies entirely above zero. The improvement is real, so the team ships B. Note that the two individual CIs barely overlap, which is easy to misread, the correct and only test is the interval on the difference, which here clears 0 decisively.
Proportions & Differences in Machine Learning & AI
Comparing models, prompts, or product variants is almost always a comparison of proportions or means, and the CI on the difference is how you decide whether one truly beats another.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| CI for a proportion | Interval on a rate metric | click-through rate, win rate, pass@1 |
| Difference of proportions | Online A/B test of two systems | does the new model lift conversion? |
| Difference of means | Comparing two models' scores | mean reward / accuracy of A vs B |
| Excludes 0? | Is the improvement real? | ship only if the lift CI clears zero |
Every "our model beats the baseline" claim is a difference of proportions or means, and it is only credible if the confidence interval on the difference excludes zero. Online A/B tests of recommender and ranking systems are exactly the difference-of-proportions interval from this chapter, run at scale. The most common error in model comparison is reading two overlapping intervals instead of building the single interval on the gap, which is both more powerful and the statistically correct test.
Build proportion and difference CIs in Python
The companion notebook builds a CI for one proportion (with the success-failure check), for a difference of
proportions, and for a difference of means, then loads confidence-intervals-for-proportions-and-differences--ab_test.xlsx to settle a real A/B
test by asking whether the lift's interval excludes zero.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓CI for a proportion: p̂ ± z·√(p̂(1−p̂)/n), valid under the success-failure condition (≥10 each).
- ✓CI for a difference of proportions: combine both SEs; the difference is real if the interval excludes 0.
- ✓CI for a difference of means: the two-sample t-interval (Welch); same exclude-0 rule.
- ✓Test the difference, not two separate intervals, overlapping individual CIs can still have a difference that clears 0.
- ✓Real A/B test: B converts 14.6% vs A 11.0%, a +3.7-point lift with a 95% CI of [+0.8, +6.6] that excludes 0, so ship B.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy and SciPy before checking the solutions.
CI for a proportion
220 of 1,000 surveyed customers churned. Build a 95% CI for the churn rate.
Success-failure condition
For 6 successes out of 40, check whether the normal-approximation CI is appropriate.
Difference of two proportions
Group A: 90/600 converted. Group B: 120/620. Build a 95% CI for pB − pA and check whether it excludes 0.
Difference of two means
Two groups spend Normal(40, 12), n = 200 and Normal(44, 13), n = 210. Build a 95% CI for the difference of means.
Real data: A/B lift
Load confidence-intervals-for-proportions-and-differences--ab_test.xlsx and report each variant's conversion CI and the CI for the lift
(B − A).
groupby("variant"), then the difference interval.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on intervals for proportions and differences. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Every interval here has a width, the margin of error. the Margin of Error chapter zooms in on the margin of error itself, what drives it, how it shrinks with sample size, and how to report a poll the right way.