←Back to the bookChapter 87 · Case Study: Comparing Marketing Channels

▶ Open in Colab⬇ Download notebookView on GitHub

Chapter 87 · Case Study · Inference

Case Study: Comparing Marketing Channels 📣

Four acquisition channels, one numeric outcome, and a budget on the line. We explore the data, then let statsmodels run the ANOVA and Tukey HSD, no hand-coded post-hoc loop, and cross-check with a rank-based test.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

⚙️ Setup

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
# statsmodels = the R/SAS-style stats library (pre-installed on Colab): it does the standard errors,
# test statistics, intervals, and post-hoc comparisons for us, so we write far less by-hand code.
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep, proportion_confint
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
from scipy.stats.contingency import association   # Cramer's V in one call
CY="#0891b2"; DEEP="#0e7490"; LIGHT="#67e8f9"; INK="#1a2138"; GRID="#e6e9f2"; GREEN="#059669"; RED="#ef4444"; AMBER="#d97706"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
pd.set_option("display.width",120)
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng = np.random.default_rng(85)

STEP 1 · EXPLORE THE DATA

Summaries and shape, per channel, before any test

First get a feel: how big is each channel group, what does revenue look like, is it skewed, are there outliers? A right-skewed revenue distribution would steer us toward a rank-based cross-check, so we look before we leap.

try:    d = pd.read_excel("../../data/case-study-comparing-marketing-channels--marketing_channels.xlsx", sheet_name="Customers")
except FileNotFoundError: d = pd.read_excel(BASE+"case-study-comparing-marketing-channels--marketing_channels.xlsx", sheet_name="Customers")
print("shape:", d.shape, "| missing:", d.isna().sum().sum())
summ = d.groupby("channel").revenue_per_customer.agg(["size","mean","std","median",
          ("skew", lambda s: stats.skew(s))]).round(1)
print(summ.sort_values("mean", ascending=False))

shape: (280, 4) | missing: 0
             size  mean   std  median  skew
channel                                    
Email          70  92.2  29.8    92.0  -0.1
Organic        70  77.6  24.8    78.9  -0.0
Paid Search    70  69.9  25.6    71.6   0.2
Social         70  67.3  25.7    69.8  -0.2

fig,ax=plt.subplots(figsize=(7,3.1))
ax.hist(d.revenue_per_customer, bins=24, color=LIGHT, alpha=0.85)
ax.axvline(d.revenue_per_customer.mean(), color=DEEP, lw=2, label=f"overall mean ${d.revenue_per_customer.mean():.0f}")
ax.set_xlabel("90-day revenue per customer ($)"); ax.set_title("Revenue distribution (all channels)"); ax.legend()
plt.tight_layout(); plt.show()

The groups are balanced (about 70 each), revenue is only mildly skewed, and Email has the highest average. ANOVA's conditions look fine, and we will still cross-check with a rank-based test because revenue can have a tail.

STEP 2 · CHOOSE THE TEST & STATE THE HYPOTHESES

A numeric outcome across four groups

Numeric outcome, FOUR independent groups. Six pairwise t-tests would inflate the false-alarm rate, so we use ONE-WAY ANOVA as the omnibus test, with Tukey HSD as the post-hoc to find which channels differ.

print("Decision: numeric outcome + 4 independent groups -> ONE-WAY ANOVA (+ Tukey HSD)")
print("H0: mu_Organic = mu_Paid = mu_Email = mu_Social  (all channels equal)")
print("H1: at least one channel mean differs")

Decision: numeric outcome + 4 independent groups -> ONE-WAY ANOVA (+ Tukey HSD)
H0: mu_Organic = mu_Paid = mu_Email = mu_Social  (all channels equal)
H1: at least one channel mean differs

STEP 3 · RUN THE ANALYSIS

statsmodels: one ANOVA table, one Tukey call

We fit the model with a formula (just like R: revenue ~ channel), read the ANOVA table from statsmodels, compute eta-squared from its sums of squares, cross-check with Kruskal-Wallis, and get every pairwise comparison from pairwise_tukeyhsd, no hand-coded loops.

# CHECK ANOVA ASSUMPTIONS before trusting the F-test (independence holds: one row per customer).
from scipy.stats import levene, shapiro
from statsmodels.stats.oneway import anova_oneway
grps  = [g.revenue_per_customer.values for _, g in d.groupby("channel")]
resid = np.concatenate([x - x.mean() for x in grps])   # pooled within-group residuals
lev = levene(*grps); sh = shapiro(resid)
print(f"Levene equal-variance   p = {lev.pvalue:.3f}  -> "
      + ("no clear evidence of unequal variances" if lev.pvalue>0.05 else "variances differ; use Welch ANOVA"))
print(f"Shapiro residual-normal p = {sh.pvalue:.3f}  -> "
      + ("residuals look normal" if sh.pvalue>0.05 else "non-normal; lean on the large n or use Kruskal-Wallis"))
w = anova_oneway(d.revenue_per_customer, d.channel, use_var="unequal")   # Welch ANOVA: no equal-variance assumption
print(f"Welch ANOVA cross-check F = {w.statistic:.2f} on (3, {w.df_denom:.1f}) df, p = {w.pvalue:.1e}")
fig, ax = plt.subplots(figsize=(4.6,3.8)); stats.probplot(resid, dist="norm", plot=ax)
ax.set_title("Residual QQ plot (points on the line = normal)"); plt.tight_layout(); plt.show()

Levene equal-variance   p = 0.374  -> no clear evidence of unequal variances
Shapiro residual-normal p = 0.106  -> residuals look normal
Welch ANOVA cross-check F = 10.89 on (3, 153.0) df, p = 1.6e-06

Before reading the F-test we confirm its assumptions, a p-value is only trustworthy if the test's conditions hold. Independence is by design (one row per customer). Levene's test (p = 0.37) finds no evidence that the channels' revenue variances differ, and the residual QQ plot with Shapiro (p = 0.11) shows near-normal residuals, so the classic ANOVA is reasonable here, and with 70 customers per channel it tolerates modest departures. Welch's ANOVA (statsmodels' anova_oneway(..., use_var="unequal"), run above), which drops the equal-variance assumption, agrees (F ≈ 10.9, p about 2e-6). For heavy non-normality, the rank-based Kruskal-Wallis test (run just below, and it agrees). We also report the effect size eta-squared next to the p-value, because a significant result is not automatically a large one.

model = ols("revenue_per_customer ~ C(channel)", data=d).fit()
aov = sm.stats.anova_lm(model, typ=2)
eta2 = aov.loc["C(channel)","sum_sq"] / aov["sum_sq"].sum()
print(aov.round(3))
print(f"\neta^2 = {eta2:.3f} (channel explains ~{eta2*100:.0f}% of revenue variation)")
groups=[g.revenue_per_customer.values for _,g in d.groupby("channel")]
print(f"Kruskal-Wallis cross-check: H={stats.kruskal(*groups).statistic:.2f}, p={stats.kruskal(*groups).pvalue:.2e} (agrees)")

                sum_sq     df      F  PR(>F)
C(channel)   26362.164    3.0  12.44     0.0
Residual    194955.764  276.0    NaN     NaN

eta^2 = 0.119 (channel explains ~12% of revenue variation)
Kruskal-Wallis cross-check: H=29.72, p=1.58e-06 (agrees)

# Tukey HSD: one function call replaces the whole pairwise loop
tukey = pairwise_tukeyhsd(d.revenue_per_customer, d.channel)
print(tukey.summary())
fig,ax=plt.subplots(figsize=(7.4,3.6))
order=d.groupby("channel").revenue_per_customer.mean().sort_values(ascending=False).index.tolist()
bp=ax.boxplot([d[d.channel==c].revenue_per_customer.values for c in order], tick_labels=order, patch_artist=True)
for patch,c in zip(bp["boxes"],[CY,LIGHT,LIGHT,LIGHT]): patch.set_facecolor(c); patch.set_alpha(0.6)
ax.set_ylabel("90-day revenue per customer ($)"); ax.set_title("Revenue by acquisition channel (one-way ANOVA)")
plt.tight_layout(); plt.show()

      Multiple Comparison of Means - Tukey HSD, FWER=0.05       
================================================================
   group1      group2   meandiff p-adj   lower    upper   reject
----------------------------------------------------------------
      Email     Organic -14.6054  0.007 -26.2172  -2.9937   True
      Email Paid Search -22.3427    0.0 -33.9545  -10.731   True
      Email      Social -24.9199    0.0 -36.5316 -13.3081   True
    Organic Paid Search  -7.7373  0.314  -19.349   3.8745  False
    Organic      Social -10.3144 0.1014 -21.9262   1.2973  False
Paid Search      Social  -2.5771 0.9399 -14.1889   9.0346  False
----------------------------------------------------------------

The ANOVA table is decisive (F ≈ 12.4, p ≈ 10⁻⁷), channel explains about 12% of revenue variation, and Kruskal-Wallis agrees. The Tukey table tells the whole story in one glance: Email differs significantly from each of the other three (reject = True), while Organic, Paid Search, and Social do not differ from one another (reject = False).

📋 STATISTICIAN’S REPORT

Recommendation: shift budget toward Email

What we found. Customers acquired through Email are worth about 92 dollars in their first 90 days, roughly 15 to 25 dollars more than those from Organic (78 dollars), Paid Search (70), or Social (67). The data cannot tell the other three channels apart.

How confident are we? If the channels truly brought in equal revenue, a spread this large would turn up far less than once in a million samples (p ≈ 0.0000001), and a rank-based method agrees, so it does not depend on assuming a perfect bell curve. The Tukey comparison (which corrects for testing several pairs) confirms Email stands apart from all three others.

What to do. Re-weight acquisition spend toward Email; don't rank Organic, Paid, and Social against each other until more data can separate them.

Caveats. This is revenue per acquired customer, not per dollar spent, fold in each channel's acquisition cost before finalizing budget. And because channels were not randomly assigned (observational data), some of Email's edge could reflect who self-selects into it.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 87 · Case Study: Comparing Marketing Channels