Case Study: A Paired Pre/Post Study
A team redesigned the checkout flow and wants to know: is the new design faster to complete? They could split users into two groups (one per design), but people vary enormously in how fast they work, and that variation would drown out the effect. Instead every user does the task on both designs, so each person serves as their own control. The analysis then works on within-person differences, and the star test is the paired t-test, cross-checked by the rank-based Wilcoxon signed-rank test.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
EM="#0891b2"; DEEP="#0e7490"; LIGHT="#a5f3fc"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; GREEN="#059669"; GREY="#94a3b8"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
try: df = pd.read_excel("../../data/case-study-a-paired-pre-post-study--ux_redesign.xlsx", sheet_name="Data")
except FileNotFoundError: df = pd.read_excel(BASE+"case-study-a-paired-pre-post-study--ux_redesign.xlsx", sheet_name="Data")
print(f"users: {len(df)} (each timed the same task on the OLD and NEW design)")
df.head(4)
users: 45 (each timed the same task on the OLD and NEW design)
| user_id | first_design | old_design_seconds | new_design_seconds | |
|---|---|---|---|---|
| 0 | U01 | old | 73.5 | 74.5 |
| 1 | U02 | old | 66.3 | 56.7 |
| 2 | U03 | new | 65.4 | 52.1 |
| 3 | U04 | new | 40.7 | 34.3 |
old, new = df.old_design_seconds, df.new_design_seconds
fig, ax = plt.subplots(1, 2, figsize=(11,4))
# left: the two distributions overlap a lot (big between-user spread)
ax[0].hist(old, bins=12, alpha=0.6, color=GREY, label=f"old (mean {old.mean():.1f}s)")
ax[0].hist(new, bins=12, alpha=0.6, color=EM, label=f"new (mean {new.mean():.1f}s)")
ax[0].set(title="The two designs overlap heavily", xlabel="task time (s)"); ax[0].legend()
# right: connect each user's two times -> almost every line slopes down
for _, r in df.iterrows():
ax[1].plot([0,1], [r.old_design_seconds, r.new_design_seconds], color=EM, alpha=0.35, lw=1)
ax[1].plot([0,1], [old.mean(), new.mean()], color=RED, lw=3, marker="o", label="group mean")
ax[1].set(title="Per user: old -> new (down = faster)", xticks=[0,1], xlim=(-0.2,1.2), ylabel="task time (s)"); ax[1].set_xticklabels(["old","new"]); ax[1].legend()
plt.tight_layout(); plt.show()
The histograms on the left overlap almost completely, because users differ so much in baseline speed that the two designs look nearly the same in aggregate. But the connected lines on the right tell the real story: most users (35 of 45) get faster on the new design. The pairing makes a consistent within-user improvement visible that the overlapping group distributions hide.
between_user_sd = df[["old_design_seconds","new_design_seconds"]].mean(axis=1).std(ddof=1)
diff = new - old
print(f"spread BETWEEN users (avg speed) : SD = {between_user_sd:.1f} s <- huge, and irrelevant to the question")
print(f"spread of the within-user DIFFERENCES : SD = {diff.std(ddof=1):.1f} s <- what the paired test actually uses")
print("Pairing subtracts each user's own baseline, so the big between-user spread never enters the test.")
spread BETWEEN users (avg speed) : SD = 16.3 s <- huge, and irrelevant to the question spread of the within-user DIFFERENCES : SD = 10.1 s <- what the paired test actually uses Pairing subtracts each user's own baseline, so the big between-user spread never enters the test.
Users' average speeds vary with a standard deviation of about 16 seconds, far bigger than the roughly 9-second effect we are hunting. A between-groups test has to see the signal through all that noise. The paired approach sidesteps it entirely by working on each user's own change, whose spread is much smaller. That is the whole reason to pair.
print(f"mean difference (new - old) = {diff.mean():+.2f} s (negative = faster on the new design)")
sh = stats.shapiro(diff)
print(f"Shapiro-Wilk on the differences: W = {sh.statistic:.3f}, p = {sh.pvalue:.3f} -> "
+ ("NON-normal" if sh.pvalue<0.05 else "roughly normal"))
fig, ax = plt.subplots(figsize=(7,3.4))
ax.hist(diff, bins=15, color=EM, alpha=0.8, edgecolor=DEEP)
ax.axvline(0, color=INK, lw=1); ax.axvline(diff.mean(), color=RED, ls="--", lw=1.5, label=f"mean {diff.mean():.1f}s")
ax.set(title="Per-user differences: two clusters, not one bell", xlabel="new - old (seconds)"); ax.legend()
plt.tight_layout(); plt.show()
mean difference (new - old) = -9.29 s (negative = faster on the new design) Shapiro-Wilk on the differences: W = 0.934, p = 0.013 -> NON-normal
The differences are not a clean bell: there is a big cluster of users who got much faster and a second cluster near zero who barely changed. Shapiro-Wilk confirms it (p = 0.01, non-normal). That matters, because the paired t-test assumes roughly normal differences. With 45 pairs the central limit theorem keeps the t-test reasonable, but a bimodal shape like this calls for cross-checks: the rank-based Wilcoxon signed-rank test, which assumes roughly symmetric differences, and the sign test, which assumes neither.
t = stats.ttest_rel(new, old)
# confidence interval for the mean difference (paired)
sd = diff.std(ddof=1); n = len(diff); se = sd/np.sqrt(n)
crit = stats.t.ppf(0.975, n-1); lo, hi = diff.mean()-crit*se, diff.mean()+crit*se
print(f"paired t-test: t({n-1}) = {t.statistic:.2f}, p = {t.pvalue:.2e}")
print(f"mean difference = {diff.mean():.2f} s 95% CI [{lo:.2f}, {hi:.2f}]")
paired t-test: t(44) = -6.17, p = 1.92e-07 mean difference = -9.29 s 95% CI [-12.33, -6.25]
The paired t-test is emphatic, t(44) = -6.17, p about 2e-7, with a 95% confidence interval on the mean difference that sits well below zero (roughly -12 to -6 seconds). Because the differences are non-normal we treat this as an approximation, but with 45 users the Central Limit Theorem keeps it reasonable, and it agrees with the robust test next.
w = stats.wilcoxon(new, old)
print(f"Wilcoxon signed-rank: W = {w.statistic:.0f}, p = {w.pvalue:.2e}")
n_faster = int((diff < 0).sum()); n_slower = int((diff > 0).sum())
print(f"{n_faster} of {len(df)} users were faster on the new design, {n_slower} were slower.")
print("Same verdict as the t-test. It drops the normality assumption but assumes roughly symmetric differences.")
Wilcoxon signed-rank: W = 116, p = 6.00e-06 35 of 45 users were faster on the new design, 10 were slower. Same verdict as the t-test. It drops the normality assumption but assumes roughly symmetric differences.
Wilcoxon ranks the absolute differences and asks whether the positives and negatives balance out. They do not: the improvements dominate (p about 6e-6), and it needs no bell-curve assumption. It does assume the differences are roughly symmetric, which a bimodal shape strains, so the sign test in the Take It Further notebook, which counts only who got faster, is the check that assumes neither. That all three agree here is what makes the verdict safe.
dz = diff.mean() / diff.std(ddof=1) # Cohen's dz for a paired design
print(f"mean improvement = {abs(diff.mean()):.1f} s faster (about {abs(diff.mean())/old.mean()*100:.0f}% of the old time)")
print(f"Cohen's dz = {dz:.2f} (paired effect size; |0.2| small, |0.5| medium, |0.8| large)")
d_av = diff.mean() / np.sqrt((old.var(ddof=1) + new.var(ddof=1)) / 2) # against the spread of the times themselves
print(f"Cohen's d = {d_av:.2f} (same difference over the task-time SD; pairing shrinks the SD that dz divides by)")
mean improvement = 9.3 s faster (about 15% of the old time) Cohen's dz = -0.92 (paired effect size; |0.2| small, |0.5| medium, |0.8| large) Cohen's d = -0.55 (same difference over the task-time SD; pairing shrinks the SD that dz divides by)
The new design saves about 9 seconds per checkout, roughly 15% of the old time, and the paired effect size is Cohen's dz about -0.9, large on the scale of the differences. Against the spread of the task times themselves it is d about -0.55, a medium effect. Reporting the effect size beside the p-value keeps the finding honest: this is not just statistically detectable, it is big enough to matter to real users.
# Pretend we had ignored the pairing and run an independent-samples t-test instead
ind = stats.ttest_ind(new, old)
print(f"PAIRED t = {t.statistic:.2f}, p = {t.pvalue:.1e} (uses each user as their own control)")
print(f"UNPAIRED t = {ind.statistic:.2f}, p = {ind.pvalue:.3f} (treats old and new as two independent groups)")
print(f"\nSame effect, but the paired t is {abs(t.statistic)/abs(ind.statistic):.1f}x larger: pairing removed the between-user noise.")
PAIRED t = -6.17, p = 1.9e-07 (uses each user as their own control) UNPAIRED t = -2.59, p = 0.011 (treats old and new as two independent groups) Same effect, but the paired t is 2.4x larger: pairing removed the between-user noise.
Analyze the very same numbers as two independent groups and the evidence weakens sharply: t drops from -6.2 to -2.6 and the p-value climbs from 2e-7 to 0.01. Nothing about the effect changed. The unpaired test still detects it, but only through the between-user variance that pairing cancels. This is the case for within-subject designs in one comparison: when you can measure each subject twice, do, and analyze the pairs.
faster = diff[diff < -5] # clear responders
flat = diff[diff >= -5] # little or no change
print(f"responders (>5s faster): {len(faster)} users, averaging {faster.mean():.1f} s faster")
print(f"non-responders : {len(flat)} users, averaging {flat.mean():+.1f} s")
slow = diff.nlargest(2) # the two biggest slow-downs
print(" two of them got markedly slower: " + ", ".join(f"{df.user_id[i]} {v:+.1f} s" for i, v in slow.items()))
rest = flat.drop(slow.index)
print(f" the other {len(rest)} averaged {rest.mean():+.1f} s")
print("The overall -9s average blends a big improvement for most, little change for some, and two users who got slower.")
responders (>5s faster): 31 users, averaging -14.9 s faster non-responders : 14 users, averaging +3.1 s
two of them got markedly slower: U27 +19.2 s, U45 +16.9 s the other 12 averaged +0.6 s The overall -9s average blends a big improvement for most, little change for some, and two users who got slower.
The two clusters in Step 5 are real groups: most users get a large speed-up, 12 barely change, and two (U27 and U45) get markedly slower. The headline -9 seconds is an average of those two stories. Worth a follow-up: who are the non-responders, and is there a task or device where the redesign does not help? A paired study points you straight at that question, which a single group mean would have buried.
Two matter here. Order and carryover: doing the task twice means practice; the second design could look better just because users learned the task. The study counterbalanced the order (about half saw each design first, recorded in first_design), and the Take It Further notebook checks that order did not drive the result. Single task, single session: one checkout task on one day is a narrow slice; a durable claim needs more tasks and users. Neither undermines the finding, but a careful report names them.
For the product team: In a within-subject test, 45 users completed the checkout task about 9 seconds faster on the new design (roughly 15% quicker), a sizable and statistically strong improvement (Wilcoxon p about 6e-6; paired t p about 2e-7). Because each user tried both designs, the test controls for the big differences in how fast people work, and it detects the effect far more strongly than a simple two-group comparison of the same numbers (p about 0.01). One nuance: the gain is concentrated in most users, while a minority saw little change and two users got markedly slower, worth investigating before a full rollout. We recommend shipping the new design and following up on the non-responders.