Two groups was a single t-test. Real decisions usually involve more: four marketing channels, five store layouts, six suppliers. Compare them pairwise and the false alarms multiply. Analysis of variance asks the global question once, then a post-hoc test fills in the detail.
Do the four channels differ in average revenue per customer by more than chance? And if so, which channel (or channels) genuinely stands apart?
The Data & The Question
One row per acquired customer: the channel that brought them and their 90-day
revenue_per_customer. The groups are balanced (about 70 customers each), which keeps the analysis
clean.
One row per customer with channel (four groups),
revenue_per_customer, and signup_month.
The sample means: Email $92, Organic $78, Paid Search $70, Social $67. Email leads, but four sample averages always wobble apart a little even if the true channel values were identical. We need to test whether the spread among these means is bigger than the within-channel noise.
Choosing the Test & the Hypotheses
The decision map: the outcome (revenue) is numeric, and there are four independent groups. Running all six pairwise t-tests would give roughly a 26% chance of a false alarm even if nothing differed. The omnibus answer is one-way ANOVA; a significant F then earns a Tukey HSD post-hoc.
Assumption checks. ANOVA needs independent groups (true by design), roughly equal variances, and roughly normal residuals. Levene's test (p ≈ 0.37) clears equal variance and a residual QQ plot with Shapiro (p ≈ 0.11) clears normality, so the conditions hold. Had equal variance failed, the remedy is Welch's ANOVA; because revenue can be right-skewed we also run the rank-based Kruskal-Wallis test as a cross-check, and it agrees, so the conclusion does not hinge on assuming a perfect bell curve.
The Analysis & Results
The F-test answers "any difference?"; η² says how much channel matters; Kruskal-Wallis confirms; Tukey HSD says which pairs separate.
| Step | Result |
|---|---|
| One-way ANOVA | F = 12.4, p ≈ 10⁻⁷ → reject H₀ |
| Effect size | η² ≈ 0.12 (channel explains ~12% of revenue variation) |
| Kruskal-Wallis cross-check | H = 29.7, p ≈ 10⁻₆ (agrees) |
| Tukey: Email vs Organic | +$14.6, CI [+3.0, +26.2] → differs |
| Tukey: Email vs Paid / Social | +$22 / +$25 → both differ |
| Tukey: Organic vs Paid vs Social | all within noise → not distinguishable |
The F-test is decisive and Kruskal-Wallis agrees, so a real difference exists and does not depend on the normality assumption. Tukey HSD localizes it cleanly: the entire story is Email, which significantly beats each of the other three, while Organic, Paid Search, and Social are statistically interchangeable on revenue.
The Statistician's Report
How to brief the head of marketing, no F-statistics required.
Recommendation: shift budget toward Email
What we found. Customers acquired through Email are worth about $92 in their first 90 days, roughly $15 to $25 more than those from Organic ($78), Paid Search ($70), or Social ($67). Those other three channels are statistically about the same as each other.
How confident are we? The overall difference is extremely unlikely to be chance (less than 1 in a million), and a method that makes no bell-curve assumption agrees. The careful pairwise comparison (which corrects for testing several at once) confirms Email stands apart from all three others, while the gaps among Organic, Paid, and Social are within the noise.
What to do. Re-weight acquisition spend toward Email, and treat Organic, Paid, and Social as equivalent on revenue for now.
Caveats. This is revenue per acquired customer, not per dollar spent, fold in each channel's acquisition cost before finalizing the budget. And because channels were not randomly assigned (this is observational), some of Email's edge could reflect who self-selects into it rather than the channel itself.
The same one-way-ANOVA-plus-post-hoc pattern is how teams compare three or more models, prompts, or configurations, and why "A/B/n" experiments need multiplicity corrections. The omnibus-then-pairwise discipline keeps a lucky winner from being mistaken for a real one.
Run the ANOVA and Tukey HSD in Python
The companion notebook explores revenue by channel (summaries, skew, a distribution plot), then lets statsmodels run the ANOVA (ols + anova_lm) and every pairwise comparison (pairwise_tukeyhsd), no hand-coded loop, cross-checks with Kruskal-Wallis, and plots revenue by channel.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓3+ groups: use one-way ANOVA, not many t-tests, to keep the false-alarm rate honest.
- ✓Cross-check a skew-prone outcome with Kruskal-Wallis; agreement makes the conclusion robust.
- ✓Omnibus then post-hoc: a significant F (p ≈ 10⁻⁷, η² ≈ 0.12) earns a Tukey HSD to find the pairs.
- ✓Result: Email beats all three other channels; Organic, Paid, and Social are indistinguishable.
- ✓Caveats: revenue per customer (not per dollar), and observational, so confirm with cost data and watch for self-selection.
Quiz: Test Yourself
Eight quick questions on this case study. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
So far, one test per question. A Clinical Trial takes a single trial and answers three questions, paired, two-arm, and a responder rate, each needing a different test on the same patients.