Every interval so far came from a formula, and every formula came with conditions. But many useful statistics, the median, a percentile, a correlation, a ratio, have no tidy standard-error formula at all. The bootstrap sidesteps the algebra entirely by simulating.
The same three lines, resample, recompute, read off the percentiles, give a confidence interval for the mean, the median, a correlation, or anything else. No new theory required.
The Bootstrap Idea
We have only one sample, but a good sample resembles its population. So instead of drawing many samples from the population (which we cannot do), we resample from our sample with replacement, each resample the same size, and recompute the statistic each time.
In the notebook, the bootstrap distribution of the mean has a spread of 1.27, matching the formula standard error s/√n of 1.28, recovered with no algebra. The bootstrap turns "imagine resampling the population" into "actually resample the sample", which is something a computer can do trivially.
The Percentile Interval
The simplest bootstrap confidence interval is the percentile method: take the 2.5th and 97.5th percentiles of the bootstrap statistics. For the mean, this should reproduce the textbook t-interval, a reassuring sanity check.
The percentile method is the easy default, but it can be slightly off when the statistic's distribution is skewed.
The library-first, more accurate choice is BCa (bias-corrected and accelerated), one call
away in scipy.stats.bootstrap(data, statistic, method="BCa"). The companion notebook reports the BCa
interval next to the percentile one for the median salary, on this 300-row sample they match almost exactly, but
BCa is the safer default on small or heavily skewed samples.
In the notebook the bootstrap percentile interval for the mean is [47.18, 52.17] against the t-interval's [47.10, 52.24], essentially identical. A method that reproduces the formula where a formula exists is exactly the kind you can trust where one does not.
Where the Bootstrap Wins
The payoff is statistics with no standard-error formula. The median is the classic case: there is no simple algebraic SE, but the bootstrap does not care, resample, take the median, read the percentiles.
| Statistic | Formula CI? | Bootstrap? |
|---|---|---|
| Mean | yes (t-interval) | yes (and agrees) |
| Median | no simple one | yes, easily |
| Percentile (e.g. 90th) | no | yes |
| Correlation | awkward / approximate | yes |
| Ratio, trimmed mean, … | rarely | yes, same loop |
The same resample-and-recompute loop delivers a CI for any of these. This generality, introduced by Bradley Efron in 1979, is why the bootstrap is one of the most used tools in modern statistics and data science. It also has cousins: the permutation test (shuffle group labels to test a difference) and the jackknife (leave-one-out resampling).
Real-World Example: Median Salary
An HR team exports 300 employee salaries. Pay is right-skewed, a few high earners pull the mean up, so the median is the fairer "typical pay" summary. But the median has no neat CI formula, exactly the case the bootstrap was built for.
One row per employee with annual_salary, department, and
years_experience.
| Summary | Point estimate | 95% confidence interval | Method |
|---|---|---|---|
| Median salary | $107,650 | $101,050 to $111,100 | bootstrap (no formula) |
| Mean salary | $111,251 | $107,335 to $115,167 | t-interval |
The median is about $107,650, with a bootstrap 95% interval of roughly $101,000 to $111,000, a defensible "typical pay" range that no textbook formula could provide. Because the data is right-skewed (skewness 0.85), the mean ($111,251) sits above the median; reporting the median with a bootstrap interval is the honest, robust summary. The bootstrap turned an awkward statistic into a routine one.
Resampling in Machine Learning & AI
Resampling is everywhere in machine learning, from honest error bars on metrics to the ensembles that power random forests.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Bootstrap CI | Error bars on any metric | 95% CI for accuracy / F1 / AUC |
| Resampling the data | Bagging & random forests | each tree trains on a bootstrap sample |
| Permutation test | Feature importance & significance | shuffle a feature to test its effect |
| Simulation | Monte Carlo everything | uncertainty when no formula exists |
The bootstrap is the standard way to put a confidence interval on a model metric: resample the test set, recompute accuracy or AUC, and read the percentiles, no distributional assumptions needed. It is also the engine of bagging: every tree in a random forest is trained on a bootstrap resample of the data, and permutation importance resamples by shuffling a feature to measure its contribution. When a quantity has no clean formula, which in modern ML is most of the time, resampling and simulation are how you quantify uncertainty honestly.
Bootstrap any statistic in Python
The companion notebook writes a three-line bootstrap, recovers the standard error and percentile interval for
the mean (matching the t-interval), bootstraps the median and a percentile with no formula, and loads
resampling-and-simulation--salaries.xlsx to put a 95% interval on the median salary.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓The bootstrap resamples the data with replacement and recomputes the statistic; the spread is the SE.
- ✓Percentile CI: the 2.5th and 97.5th percentiles of the bootstrap statistics form a 95% interval.
- ✓It reproduces the t-interval for the mean and works for statistics with no formula (median, percentile, correlation).
- ✓Real data: median salary $107,650 with a bootstrap 95% interval of about $101k–$111k, where no formula exists.
- ✓In ML/AI: bootstrap CIs on metrics, bagging/random forests, and permutation importance are all resampling.
Practice Challenges
Five short challenges, beginner to intermediate. Try them with NumPy before checking the solutions.
Bootstrap the standard error
From a sample of 80 from Normal(100, 15), bootstrap the SE of the mean and compare to s/√n.
Percentile CI for the mean
Build the bootstrap 95% percentile CI for the mean and compare to the t-interval.
np.percentile(boot, [2.5, 97.5]).CI for the median
On right-skewed lognormal data (n = 150), build a bootstrap 95% CI for the median.
np.mean for np.median.CI for a correlation
Bootstrap a 95% CI for the correlation between two related variables.
Real data: median salary
Load resampling-and-simulation--salaries.xlsx and build a bootstrap 95% CI for the median salary.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on resampling and the bootstrap. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now turn a sample into a best guess and an honest interval, for means, proportions, differences, and even formula-free statistics via the bootstrap, and report the margin of error correctly. Hypothesis Testing & Inference takes the mirror image: instead of estimating a value, we test a claim, starting with the logic of hypothesis testing.