Every "we shipped the new design and conversion went up" decision at a modern company is, underneath, a hypothesis test. The A/B test is the controlled experiment from the Study Design & Data Quality chapter run online at scale, and getting it right means combining nearly everything in this Part: randomization, power, the two-proportion z-test, confidence intervals, and the discipline not to fool yourself.
Randomize, power it first, analyze the lift with a CI (not just a p-value), guard your secondary metrics, and do not peek. Significance plus a meaningful effect size equals ship.
Anatomy of an A/B Test
Randomization is the heart of it: assigning each user to A or B by a coin flip makes the two groups statistically identical on everything except the change you are testing, so any difference in the metric is caused by the change (the logic of the Study Design & Data Quality chapter, online). Before launch you fix one primary metric and a decision rule.
The hypotheses write themselves: H₀ is "no difference" (p₊ = p₀), H₁ is "B differs" (usually "B is better"). The decision rule is set in advance, ship B if the lift's confidence interval clears zero (and the effect is large enough to be worth it). Everything else is execution.
Powering the Test: Decide n First
The single most important discipline: choose the sample size before launch from a power analysis. You specify the baseline rate, the minimum lift worth detecting (the MDE), α, and the target power (usually 80%), and solve for n per arm.
The notebook computes these directly: to catch a +1-point lift off a 12% baseline you need about 17,000 users per arm, but a +3-point lift needs only about 2,000. Because n grows like 1/MDE², halving the lift you want to detect roughly quadruples the sample. Fixing n in advance is exactly what makes the 5% false-positive rate and the 80% power real.
Analyzing the Result, and the Peeking Trap
The analysis is the two-proportion z-test from the z-Tests chapter, reported with a confidence interval for the lift. The CI does double duty: if it excludes 0 the effect is real, and its width shows how big the lift plausibly is, so you can judge whether it is worth shipping.
The cardinal sin is peeking: checking the results repeatedly and stopping the instant p < 0.05. The notebook shows it turns a 5% test into roughly a 25% false-positive rate on an A/A test with no real difference, you simply give chance many shots at the threshold. The fixes are to analyze once at the pre-set n, or to use proper sequential methods (alpha-spending, always-valid p-values) built for continuous monitoring.
Real-World Example: A Landing-Page Experiment
A team ran a randomized A/B test on a redesigned landing page, 4,200 sessions split between control (A) and the new design (B). We analyze the primary metric (conversion) and a secondary metric (revenue per session), then make the ship decision.
One row per session with variant (A/B), converted (0/1),
revenue, and device.
| Metric | A (control) | B (new design) | Test | Result |
|---|---|---|---|---|
| Conversion (primary) | 11.0% | 13.3% | two-proportion z | lift +2.25 pts, 95% CI [+0.3, +4.2], p ≈ 0.026 |
| Revenue / session (guardrail) | $5.95 | $7.34 | Welch t | p ≈ 0.017 (also up) |
| Decision | Ship B, both metrics improved, lift CI clears zero, n fixed in advance | |||
The verdict is to ship B. Conversion rises from 11.0% to 13.3%, a lift of about +2.25 points whose 95% interval (roughly +0.3 to +4.2 points) sits entirely above zero (z = 2.23, p ≈ 0.026). The guardrail metric agrees, revenue per session climbs from about $5.95 to $7.34 (Welch p ≈ 0.017), so the conversion win is not cannibalizing order value. Both metrics point the same way, and because the sample size was fixed in advance, the 5% guarantee is intact. This is a clean, defensible, shippable result, the entire Part working together.
A/B Testing in Machine Learning & AI
Online experimentation is how ML systems are actually shipped and trusted in production.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| A/B test of a change | Shipping a model online | new ranking model vs the incumbent |
| Power / sample size | How long to run an experiment | days of traffic for a detectable lift |
| Guardrail metrics | Don't win one metric, hurt another | clicks up but dwell-time down? |
| Sequential / no-peeking | Multi-armed bandits, always-valid tests | adaptive traffic with valid inference |
Offline metrics rarely tell the whole story, so the gold standard for deploying a model, a recommender, a ranker, a generative feature, is an online A/B test against the current system. The same machinery applies: power the experiment for the lift you care about, analyze the primary metric with a proportion test and a CI, and protect guardrail metrics so a click-through win does not secretly tank user satisfaction. Multi-armed bandits push further, adaptively shifting traffic to better variants, but they demand sequential, always-valid inference precisely because they "peek" by design. The discipline of this chapter is what keeps "the new model is better" a claim you can stand behind.
Run an A/B test end to end in Python
The companion notebook computes the required sample size from a power analysis, analyzes the primary metric
with a two-proportion z-test and a lift CI, demonstrates how peeking inflates false positives, and loads
a-b-testing-and-online-experiments--online_experiment.xlsx to reach a ship decision on conversion and revenue.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓An A/B test randomizes users, then compares the primary rate with a two-proportion z-test and a lift CI.
- ✓Power first: fix n in advance from baseline, MDE, α, and power; n grows like 1/MDE².
- ✓Report the lift CI, not just a p-value, and protect guardrail (secondary) metrics.
- ✓Don't peek: repeated looks inflate false positives (~25% in the demo); analyze once or use sequential methods.
- ✓Real data: B lifts conversion +2.25 pts (95% CI +0.3 to +4.2, p ≈ 0.026) and revenue too, so ship B.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
Sample size for an A/B test
Baseline 12%, detect a +2-point lift at 80% power, α = 0.05. How many users per arm?
Two-proportion lift CI
A: 472/4000, B: 572/4000. Test the lift and give its 95% confidence interval.
The peeking trap
Simulate an A/A test (true rate 0.12 in both) checked 10 times; report the false-positive rate.
Secondary metric
Compare revenue per session between two simulated arms with a Welch t-test.
Real data: the experiment
Load a-b-testing-and-online-experiments--online_experiment.xlsx; run the two-proportion z-test on conversion and report the lift CI.
converted by variant.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on A/B testing. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now state a hypothesis, choose the right test, weigh both error types, and turn data into a defensible decision, from a single t-test to a full online experiment. Review & Choosing the Right Test consolidates the whole Part into one reference, before inference moves on to relationships between variables.