Chapter 75 · Estimation & Confidence Intervals
Resampling & Simulation 🔁
When no formula exists, let the data resample itself. The bootstrap builds a confidence interval for almost any statistic by drawing samples-of-the-sample. We validate it against the t-interval, then put a CI on a real MEDIAN salary, where no simple formula exists.
Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026
⚙️ Setup¶
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
BLUE="#2563eb"; DEEP="#1d4ed8"; LIGHT="#60a5fa"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; PINK="#db2777"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(73)
DEMO 1 · THE BOOTSTRAP IDEA
Resample the sample, with replacement
We only have one sample, but we can mimic 'drawing many samples' by resampling FROM our sample with replacement. Each resample is the same size; recompute the statistic each time to build its bootstrap distribution.
In [2]:
data = rng.normal(50, 10, 60) # our one sample
def bootstrap(x, stat=np.mean, B=10000):
n=len(x); idx=rng.integers(0, n, size=(B, n))
return stat(x[idx], axis=1)
boot_means = bootstrap(data, np.mean)
print(f"original sample mean = {data.mean():.2f}")
print(f"bootstrap distribution of the mean: center {boot_means.mean():.2f}, spread (= SE) {boot_means.std():.3f}")
print(f"formula SE = s/sqrt(n) = {data.std(ddof=1)/np.sqrt(60):.3f} (the bootstrap recovers it with no formula)")
original sample mean = 49.67 bootstrap distribution of the mean: center 49.68, spread (= SE) 1.270 formula SE = s/sqrt(n) = 1.284 (the bootstrap recovers it with no formula)
The spread of the bootstrap distribution is the standard error, recovered without any formula. The bootstrap turns "imagine resampling the population" into "actually resample the sample", and it works because a good sample resembles the population it came from.
DEMO 2 · THE PERCENTILE INTERVAL
Read the CI straight off the bootstrap distribution
The simplest bootstrap CI is the percentile method: take the 2.5th and 97.5th percentiles of the bootstrap statistics. For a mean, this should agree closely with the textbook t-interval, a good sanity check.
In [3]:
lo, hi = np.percentile(boot_means, [2.5, 97.5])
print(f"bootstrap 95% percentile CI for the mean: [{lo:.2f}, {hi:.2f}]")
se=data.std(ddof=1)/np.sqrt(60); t=stats.t.ppf(0.975,59)
tlo,thi = data.mean()-t*se, data.mean()+t*se
print(f"classic t-interval for the mean: [{tlo:.2f}, {thi:.2f}]")
print("-> they agree: the bootstrap reproduces the formula answer for the mean")
bootstrap 95% percentile CI for the mean: [47.18, 52.17] classic t-interval for the mean: [47.10, 52.24] -> they agree: the bootstrap reproduces the formula answer for the mean
For the mean, the bootstrap and the t-interval give essentially the same answer, exactly what we want for a method that should generalize the formula. The payoff comes when there is no formula.
DEMO 3 · WHERE THE BOOTSTRAP WINS
A CI for the median (no formula)
The median has no simple standard-error formula. The bootstrap does not care: resample, take the median each time, read off the percentiles. The same three lines that handled the mean now handle the median.
In [4]:
skewed = rng.lognormal(4, 0.5, 200) # right-skewed data
boot_med = bootstrap(skewed, np.median)
lo, hi = np.percentile(boot_med, [2.5, 97.5])
print(f"sample median = {np.median(skewed):.2f}")
print(f"bootstrap 95% CI for the MEDIAN: [{lo:.2f}, {hi:.2f}] (no formula needed)")
# the same machinery works for ANY statistic, e.g. the 90th percentile or a trimmed mean
b90 = bootstrap(skewed, lambda a,axis: np.percentile(a,90,axis=axis))
print(f"bootstrap 95% CI for the 90th percentile: [{np.percentile(b90,2.5):.1f}, {np.percentile(b90,97.5):.1f}]")
sample median = 53.47 bootstrap 95% CI for the MEDIAN: [49.75, 58.99] (no formula needed) bootstrap 95% CI for the 90th percentile: [90.0, 119.6]
No new theory, no new formula, the same resample-and-recompute loop delivers a confidence interval for the median, a percentile, a correlation, a ratio, anything you can compute. This generality is why the bootstrap (Efron, 1979) is one of the most useful tools in modern statistics.
REAL-WORLD EXAMPLE · SALARIES
A confidence interval for the median salary
An HR team exports 300 employee salaries (resampling-and-simulation--salaries.xlsx). Pay is right-skewed (a few high earners), so the median is the fairer summary, but it has no neat CI formula. The bootstrap handles it directly.
In [5]:
try: emp = pd.read_excel("../../data/resampling-and-simulation--salaries.xlsx", sheet_name="Employees")
except FileNotFoundError: emp = pd.read_excel(BASE+"resampling-and-simulation--salaries.xlsx", sheet_name="Employees")
print("loaded:", emp.shape)
sal = emp["annual_salary"].values
print(f"n={len(sal)}, mean=${sal.mean():,.0f}, median=${np.median(sal):,.0f}, skew={stats.skew(sal):.2f} (right-skewed)")
loaded: (300, 4) n=300, mean=$111,251, median=$107,650, skew=0.85 (right-skewed)
In [6]:
boot_med = bootstrap(sal, np.median, B=20000)
mlo, mhi = np.percentile(boot_med, [2.5, 97.5])
print(f"BOOTSTRAP 95% CI for the MEDIAN salary: [${mlo:,.0f}, ${mhi:,.0f}] (point ${np.median(sal):,.0f})")
# compare to the normal-theory CI for the MEAN
se=sal.std(ddof=1)/np.sqrt(len(sal)); t=stats.t.ppf(0.975,len(sal)-1)
print(f"t-interval 95% CI for the MEAN salary: [${sal.mean()-t*se:,.0f}, ${sal.mean()+t*se:,.0f}] (point ${sal.mean():,.0f})")
# Library-first, and a sharper interval: scipy.stats.bootstrap returns the BCa interval,
# which corrects the plain percentile method for bias and skew (it matters for a skewed median).
from scipy.stats import bootstrap as scipy_bootstrap
bca = scipy_bootstrap((sal,), np.median, confidence_level=0.95, method="BCa", n_resamples=20000, random_state=0)
print(f"scipy BCa 95% CI for the MEDIAN salary: [${bca.confidence_interval.low:,.0f}, ${bca.confidence_interval.high:,.0f}] (BCa = bias-corrected and accelerated)")
BOOTSTRAP 95% CI for the MEDIAN salary: [$101,050, $111,100] (point $107,650) t-interval 95% CI for the MEAN salary: [$107,335, $115,167] (point $111,251)
scipy BCa 95% CI for the MEDIAN salary: [$101,050, $111,100] (BCa = bias-corrected and accelerated)
In [7]:
fig,ax=plt.subplots(1,2,figsize=(11,3.3))
ax[0].hist(sal, bins=30, color=LIGHT, alpha=0.85)
ax[0].axvline(np.median(sal),color=GREEN,lw=2,label=f"median ${np.median(sal):,.0f}")
ax[0].axvline(sal.mean(),color=PINK,lw=2,ls="--",label=f"mean ${sal.mean():,.0f}")
ax[0].set_title("Salaries are right-skewed (mean > median)"); ax[0].set_xlabel("annual salary"); ax[0].legend(fontsize=8)
ax[1].hist(boot_med, bins=40, color=BLUE, alpha=0.75)
ax[1].axvline(mlo,color=DEEP,lw=2); ax[1].axvline(mhi,color=DEEP,lw=2)
ax[1].set_title("Bootstrap median, 20,000 draws"); ax[1].set_xlabel("resampled median")
from matplotlib.ticker import FuncFormatter, MaxNLocator
for a in ax: a.xaxis.set_major_formatter(FuncFormatter(lambda v,_: f"${v/1000:.0f}k")); a.xaxis.set_major_locator(MaxNLocator(6)); a.tick_params(axis="x", labelsize=8)
plt.tight_layout(); plt.show()
The median salary is about 107,650 dollars, with a bootstrap 95% interval of roughly 101,000 to 111,000, a defensible "typical pay" range that no textbook formula could provide. Because pay is right-skewed, the mean (111,251) sits above the median; reporting the median with a bootstrap interval is the honest, robust summary. The bootstrap turned an awkward statistic into a routine one. For a defensible published number you would report the BCa interval from scipy.stats.bootstrap, which nudges the plain percentile ends to correct for the skew, here it matches the percentile interval almost exactly, because n = 300 is large; BCa is the safer default on small or very skewed samples.
✅ Resampling, and the end of Part XI
The bootstrap resamples the sample with replacement to build the distribution of any statistic; its spread is the standard error, and its 2.5th/97.5th percentiles are a 95% CI. It reproduces the t-interval for the mean and, crucially, gives a CI for statistics with no formula, like the median salary (about 108k dollars, 95% CI about 101k-111k). With estimation complete, Part XII turns to its mirror image: hypothesis testing.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher