Chapter 71 · Estimation & Confidence Intervals
CIs for Proportions & Differences 🅰️🅱️
Polls and A/B tests estimate proportions, and the interesting question is usually a DIFFERENCE. We build CIs for one proportion, for a difference of two proportions, and of two means, then settle a real A/B test from a spreadsheet.
Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026
⚙️ Setup¶
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
BLUE="#2563eb"; DEEP="#1d4ed8"; LIGHT="#60a5fa"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
z = stats.norm.ppf(0.975)
rng = np.random.default_rng(71)
DEMO 1 · CI FOR ONE PROPORTION
p-hat plus or minus a margin
For a yes/no outcome, the proportion p-hat = successes/n estimates the true rate p. Its CI is p-hat +/- z*sqrt(p-hat(1-p-hat)/n), valid when there are at least ~10 successes AND ~10 failures (the success-failure condition).
In [2]:
successes, n = 84, 500
phat = successes/n
se = np.sqrt(phat*(1-phat)/n)
lo, hi = phat - z*se, phat + z*se
print(f"p-hat = {successes}/{n} = {phat:.3f}, SE = {se:.4f}")
print(f"95% CI for the proportion: [{lo:.3f}, {hi:.3f}] = {phat*100:.1f}% +/- {z*se*100:.1f} pts")
print(f"success-failure check: {successes} successes, {n-successes} failures (both >= 10? {successes>=10 and n-successes>=10})")
p-hat = 84/500 = 0.168, SE = 0.0167 95% CI for the proportion: [0.135, 0.201] = 16.8% +/- 3.3 pts success-failure check: 84 successes, 416 failures (both >= 10? True)
The proportion is 16.8% with a 95% interval of about [13.5%, 20.1%]. The success-failure condition (≥10 of each) is what lets us use the normal approximation; with very few successes, prefer the Wilson or exact interval instead.
DEMO 2 · CI FOR A DIFFERENCE OF PROPORTIONS
Does group B beat group A?
The real question in an experiment is the DIFFERENCE pB - pA. Its standard error combines both groups: SE = sqrt(pA(1-pA)/nA + pB(1-pB)/nB). If the CI for the difference excludes 0, the groups differ.
In [3]:
pA, nA = 0.12, 800
pB, nB = 0.155, 820
diff = pB - pA
se = np.sqrt(pA*(1-pA)/nA + pB*(1-pB)/nB)
lo, hi = diff - z*se, diff + z*se
print(f"pA = {pA:.3f}, pB = {pB:.3f}, difference pB-pA = {diff:+.3f}")
print(f"SE of difference = {se:.4f}")
print(f"95% CI for (pB - pA): [{lo:+.3f}, {hi:+.3f}]")
print("-> excludes 0: the difference is statistically distinguishable from zero" if lo>0 or hi<0 else "-> includes 0: cannot rule out no difference")
pA = 0.120, pB = 0.155, difference pB-pA = +0.035 SE of difference = 0.0171 95% CI for (pB - pA): [+0.002, +0.068] -> excludes 0: the difference is statistically distinguishable from zero
Because the interval for the difference lies entirely above 0, B's higher rate is unlikely to be a fluke. This is the engine of A/B testing: build a CI for the lift, and act if it clears 0.
DEMO 3 · CI FOR A DIFFERENCE OF MEANS
The same idea for numeric outcomes
When the outcome is numeric (revenue per user, time on site), compare two means with a two-sample t-interval: (xB - xA) +/- t* sqrt(sA^2/nA + sB^2/nB).
In [4]:
a = rng.normal(52, 14, 240) # group A spend
b = rng.normal(56, 15, 260) # group B spend
diff = b.mean() - a.mean()
se = np.sqrt(a.var(ddof=1)/len(a) + b.var(ddof=1)/len(b))
dfw = (se**4) / ((a.var(ddof=1)/len(a))**2/(len(a)-1) + (b.var(ddof=1)/len(b))**2/(len(b)-1)) # Welch df
tcrit = stats.t.ppf(0.975, dfw)
print(f"mean A = {a.mean():.2f}, mean B = {b.mean():.2f}, difference = {diff:+.2f}")
print(f"95% CI for (meanB - meanA): [{diff-tcrit*se:+.2f}, {diff+tcrit*se:+.2f}] (Welch df={dfw:.0f})")
mean A = 51.07, mean B = 56.16, difference = +5.09 95% CI for (meanB - meanA): [+2.80, +7.38] (Welch df=494)
Same logic, numeric flavor: the two-sample t-interval estimates the gap between group means and its uncertainty. Whether the CI excludes 0 tells you if the difference is real.
REAL-WORLD EXAMPLE · A/B TEST
A control vs a new landing page
A product team ran a randomized A/B test and exported 2,000 visitors (confidence-intervals-for-proportions-and-differences--ab_test.xlsx): variant A is the control, B is a new design. We estimate each conversion rate with a CI, then build the CI for the lift.
In [5]:
try: ab = pd.read_excel("../../data/confidence-intervals-for-proportions-and-differences--ab_test.xlsx", sheet_name="Visitors")
except FileNotFoundError: ab = pd.read_excel(BASE+"confidence-intervals-for-proportions-and-differences--ab_test.xlsx", sheet_name="Visitors")
print("loaded:", ab.shape)
summary = ab.groupby("variant")["converted"].agg(["size","sum","mean"]).rename(columns={"size":"n","sum":"conversions","mean":"rate"})
print(summary.round(4))
loaded: (2000, 6)
n conversions rate
variant
A 995 109 0.1095
B 1005 147 0.1463
In [6]:
def prop_ci(succ, n):
p = succ/n; se = np.sqrt(p*(1-p)/n); return p, p-z*se, p+z*se
gA = ab[ab.variant=="A"]; gB = ab[ab.variant=="B"]
nA, sA = len(gA), gA.converted.sum(); nB, sB = len(gB), gB.converted.sum()
pA, loA, hiA = prop_ci(sA, nA); pB, loB, hiB = prop_ci(sB, nB)
print(f"A (control): {pA*100:.2f}% 95% CI [{loA*100:.2f}%, {hiA*100:.2f}%] (n={nA})")
print(f"B (new design): {pB*100:.2f}% 95% CI [{loB*100:.2f}%, {hiB*100:.2f}%] (n={nB})")
diff = pB - pA; se = np.sqrt(pA*(1-pA)/nA + pB*(1-pB)/nB)
lo, hi = diff - z*se, diff + z*se
print(f"\nLIFT (B - A) = {diff*100:+.2f} pts")
print(f"95% CI for the lift: [{lo*100:+.2f} pts, {hi*100:+.2f} pts]")
print("-> CI excludes 0: ship B, the lift is real" if lo>0 else "-> CI includes 0: inconclusive")
A (control): 10.95% 95% CI [9.01%, 12.90%] (n=995) B (new design): 14.63% 95% CI [12.44%, 16.81%] (n=1005) LIFT (B - A) = +3.67 pts 95% CI for the lift: [+0.75 pts, +6.59 pts] -> CI excludes 0: ship B, the lift is real
In [7]:
fig,ax=plt.subplots(figsize=(6.5,3.2))
ax.errorbar([pA,pB],[1,0],xerr=[[pA-loA,pB-loB],[hiA-pA,hiB-pB]],fmt="o",color=BLUE,capsize=7,lw=2,ms=9)
ax.set_yticks([1,0]); ax.set_yticklabels(["A (control)","B (new design)"]); ax.set_ylim(-0.6,1.6)
ax.set_xlabel("conversion rate"); ax.set_title("Conversion CIs: B sits clearly above A")
plt.tight_layout(); plt.show()
Variant A converts at about 11.0% and B at 14.6%, a lift of roughly +3.7 points whose 95% interval runs about [+0.8, +6.6] points, entirely above zero. The new design is a genuine improvement, not noise, so the team ships B. Notice the two individual CIs barely overlap; the proper test is always the CI on the difference, not eyeballing two separate intervals.
✅ Proportions & differences
A proportion CI is p-hat +/- z*sqrt(p-hat(1-p-hat)/n), valid under the success-failure condition. For a difference, combine the two standard errors and check whether the CI excludes 0. On a real A/B test, B converts 14.6% vs A 11.0%, a +3.7 point lift with a 95% CI of about [+0.8, +6.6] that clears zero, so B wins.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher