This is the first of five case studies that put the whole inference toolkit to work. Each takes one real dataset from a fuzzy business question to a confident recommendation. We start where most data teams spend their days: an A/B test.
Did the redesign (B) convert at a higher rate than the control (A), by more than random chance can explain? And if so, how big is the gain, and is it worth shipping?
The Data & The Question
The export has one row per session: which variant the visitor saw and whether they
converted. Because assignment was random, the two groups are comparable on everything else, so any
real difference in conversion is caused by the page.
One row per session with variant (A/B), converted (0/1),
device, and seconds_on_page.
The raw rates: about 1,830 sessions saw A and converted at 9.9%; about 1,770 saw B and converted at 13.4%. B is ahead, but a raw gap is a starting point, not a verdict, random noise alone produces some difference even when nothing changed. That is exactly what a hypothesis test settles.
Choosing the Test & the Hypotheses
Run the decision map. The outcome, converted, is a yes/no proportion. There are two independent groups (A and B). That points squarely at the two-proportion z-test. Because we specifically expect B to be better, the alternative is one-sided.
Assumption checks. The z-test for proportions needs at least ~10 conversions and ~10 non-conversions per arm; here we have hundreds of each, so the normal approximation is rock solid. A quick power sanity-check confirms that with ~1,800 sessions per arm and a ~10% baseline, a lift of a few points is easily detectable, so even a null result would have meant something.
The Analysis & Results
We compute the z statistic with the pooled standard error, its p-value, and a 95% confidence interval for the lift (the unpooled SE). The interval is what makes the result actionable.
| Quantity | Value |
|---|---|
| Conversion A / B | 9.9% / 13.4% |
| Absolute lift (B − A) | +3.5 percentage points |
| Relative lift | +35% |
| Test statistic | z = 3.28 |
| p-value (one-sided) | ≈ 0.0005 |
| 95% CI for the lift | [+1.4, +5.6] pts — excludes 0 |
| Decision | reject H₀: B converts significantly better |
The lift is about +3.5 points (a ~35% relative gain), the one-sided p-value is around 0.0005, and the 95% interval for the lift, roughly +1.4 to +5.6 points, lies entirely above zero. Even the cautious end of that interval is a meaningful improvement.
The Statistician's Report
Here is how you would write this up for a product manager who does not care about z-scores, only about the decision.
Recommendation: ship the redesign
What we found. The redesigned page converted at 13.4% versus 9.9% for the current page, a gain of about 3.5 percentage points, or roughly a 35% relative increase in conversions.
How confident are we? Very. If the redesign truly made no difference, a gap this large would appear by chance only about 1 in 2,000 times (p ≈ 0.0005). Our best estimate of the true gain is a range of +1.4 to +5.6 points (95% confidence), so even the pessimistic end is a clear win.
What to do. Roll the redesign out to all traffic. Because visitors were randomly assigned, the improvement is caused by the page itself, not by who happened to see it.
One caveat. This measures conversion during the test window only. After launch, keep an eye on revenue per order and any longer-term behavior to confirm the gain holds and is not just pulling conversions forward.
This is exactly what experimentation platforms (at Netflix, Booking, Microsoft, and countless startups) automate at scale, with pre-registered sample sizes, guardrail metrics, and sometimes sequential or bandit methods. The statistics underneath are the two-proportion test and the lift interval you just ran.
Work the whole A/B test in Python
The companion notebook explores the data first (size, missing values, and a randomization check on the device mix), states the hypotheses, then lets statsmodels run the two-proportion z-test (proportions_ztest) and the lift interval (confint_proportions_2indep), adds a power check, and plots the two rates.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Outcome + groups: a yes/no rate across two independent groups → the two-proportion z-test.
- ✓One-sided H₁ because we predicted the direction (B better), decided before seeing the data.
- ✓Result: +3.5 pt lift (z = 3.28, one-sided p ≈ 0.0005), 95% CI [+1.4, +5.6] pts, excludes 0.
- ✓Report the interval, not just the p, so the decision rests on the plausible size of the gain.
- ✓Recommendation: ship B, then monitor revenue and longer-term metrics as a guardrail.
Quiz: Test Yourself
Eight quick questions on this case study. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Two groups was the easy case. Comparing Marketing Channels tackles four groups at once, where a single ANOVA replaces a thicket of t-tests and Tukey HSD names the winner.