A Paired Pre/Post Study: Take It Further
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from scipy import stats
EM="#0891b2"; DEEP="#0e7490"; LIGHT="#a5f3fc"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; GREEN="#059669"; GREY="#94a3b8"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
try: df = pd.read_excel("../../data/case-study-a-paired-pre-post-study--ux_redesign.xlsx", sheet_name="Data")
except FileNotFoundError: df = pd.read_excel(BASE+"case-study-a-paired-pre-post-study--ux_redesign.xlsx", sheet_name="Data")
df["diff"] = df.new_design_seconds - df.old_design_seconds
rng = np.random.default_rng(138)
print(f"mean difference (new - old) = {df['diff'].mean():.2f} s over {len(df)} users")
mean difference (new - old) = -9.29 s over 45 users
The study counterbalanced which design each user saw first, precisely so that practice with the task could not masquerade as a design effect. But counterbalancing is a safeguard, not a guarantee, so we test it: does the improvement differ between users who saw the old design first and those who saw the new one first? If order were driving the result, those two groups would show very different differences.
g = df.groupby("first_design")["diff"]
print(g.agg(['size','mean']).round(2))
a = df[df.first_design=='old']['diff']; b = df[df.first_design=='new']['diff']
tt = stats.ttest_ind(a, b); mw = stats.mannwhitneyu(a, b)
print(f"\nimprovement by first-seen design: t-test p = {tt.pvalue:.2f}, Mann-Whitney p = {mw.pvalue:.2f}")
print("-> no significant difference: order did not drive the effect, so counterbalancing did its job.")
size mean first_design new 22 -8.64 old 23 -9.91 improvement by first-seen design: t-test p = 0.68, Mann-Whitney p = 0.46 -> no significant difference: order did not drive the effect, so counterbalancing did its job.
The average improvement is about the same whether a user started on the old or the new design (-9.9 vs -8.6 s), and neither a t-test nor its rank-based cousin finds a difference. Half that gap, about 0.6 s, estimates the practice effect of a second try, and because the orders were split almost evenly, it cancels out of the overall mean difference. Had the two groups diverged sharply, that would point to a large order effect worth understanding, though counterbalancing would still keep it out of the average.
The paired t-test assumes normal differences; Wilcoxon only assumes a symmetric-ish distribution and uses ranks; the sign test assumes nothing at all, it just counts how many users got faster and tests that against a coin flip. Running all three shows how the conclusion survives as you strip away assumptions, the hallmark of a robust result.
d = df['diff'].values; n = len(d)
t = stats.ttest_rel(df.new_design_seconds, df.old_design_seconds)
w = stats.wilcoxon(d)
n_faster = int((d < 0).sum())
sign = stats.binomtest(n_faster, n, 0.5, alternative='two-sided') # the sign test
print(f"paired t-test (needs normal diffs) : p = {t.pvalue:.1e}")
print(f"Wilcoxon signed-rank (ranks) : p = {w.pvalue:.1e}")
print(f"sign test ({n_faster}/{n} faster, needs nothing): p = {sign.pvalue:.1e}")
print("\nAll three reject 'no change' decisively. As assumptions weaken, some power is lost, but the verdict holds.")
paired t-test (needs normal diffs) : p = 1.9e-07 Wilcoxon signed-rank (ranks) : p = 6.0e-06 sign test (35/45 faster, needs nothing): p = 2.5e-04 All three reject 'no change' decisively. As assumptions weaken, some power is lost, but the verdict holds.
Every rung of the ladder agrees: the improvement is real. The sign test throws away everything except the direction of each change (how many of the 45 users got faster) and still lands far below any reasonable threshold. When a finding survives from the assumption-heavy t-test all the way down to the assumption-free sign test, you can report it with confidence.
With non-normal differences, a bootstrap gives an assumption-light interval for the mean improvement: resample the 45 users with replacement, recompute the mean difference, and repeat. The middle 95% of those means is the interval, no t-distribution required.
boot = np.array([rng.choice(d, size=n, replace=True).mean() for _ in range(10000)])
blo, bhi = np.percentile(boot, [2.5, 97.5])
tlo, thi = stats.t.interval(0.95, n-1, loc=d.mean(), scale=stats.sem(d))
print(f"bootstrap 95% CI for the mean improvement = [{blo:.2f}, {bhi:.2f}] s")
print(f"t-based 95% CI (from the chapter) = [{tlo:.2f}, {thi:.2f}] s (they agree closely)")
fig, ax = plt.subplots(figsize=(7,3.1))
ax.hist(boot, bins=40, color=LIGHT, edgecolor=DEEP)
ax.axvline(0, color=INK, lw=1); ax.axvline(d.mean(), color=EM, lw=2, label=f'mean {d.mean():.1f}s')
for x in (blo,bhi): ax.axvline(x, color=RED, ls='--', lw=1.3)
ax.set(title='Bootstrap distribution of the mean improvement', xlabel='mean (new - old), seconds'); ax.legend()
plt.tight_layout(); plt.show()
bootstrap 95% CI for the mean improvement = [-12.13, -6.28] s t-based 95% CI (from the chapter) = [-12.33, -6.25] s (they agree closely)
The bootstrap interval sits right on top of the t-based one and, crucially, the entire interval is below zero: even the end closest to zero still has the new design several seconds faster. Two routes to the same interval, one assuming a t-distribution and one assuming nothing, is exactly the agreement you want before quoting a number.
The chapter showed that a between-groups analysis of the same data was far weaker. Here we quantify it by simulation: draw many studies at this effect size and this between-user spread, compare a paired design (same users, both conditions) against a between-subjects design (a different group of users for each design) at the same total sample size, and count how often each detects the effect. That fraction is statistical power.
eff = d.mean(); sig = d.std(ddof=1)/np.sqrt(2) # per-measurement noise (diff SD = sig*sqrt(2))
sd_between = df[['old_design_seconds','new_design_seconds']].mean(axis=1).std(ddof=1)
def one_study():
# PAIRED design: 45 users each do BOTH conditions (each is their own control)
base = rng.normal(60, sd_between, n)
p_pair = stats.ttest_rel(base + eff + rng.normal(0,sig,n), base + rng.normal(0,sig,n)).pvalue
# BETWEEN-SUBJECTS design: 45 DIFFERENT users, split into two groups (no pairing possible)
h = n//2
g_old = rng.normal(60, sd_between, h) + rng.normal(0, sig, h)
g_new = rng.normal(60, sd_between, n-h)+eff + rng.normal(0, sig, n-h)
p_bs = stats.ttest_ind(g_new, g_old).pvalue
return p_pair < 0.05, p_bs < 0.05
res = np.array([one_study() for _ in range(2000)])
print(f"power of the PAIRED design = {res[:,0].mean()*100:.0f}% (same 45 users, both designs)")
print(f"power of a BETWEEN-SUBJECTS design = {res[:,1].mean()*100:.0f}% (45 different users split into two groups)")
print("Same total sample and effect: pairing cancels the between-user spread and buys a large jump in power.")
power of the PAIRED design = 100% (same 45 users, both designs) power of a BETWEEN-SUBJECTS design = 40% (45 different users split into two groups) Same total sample and effect: pairing cancels the between-user spread and buys a large jump in power.
At this effect size and sample, the paired design detects the improvement nearly every time, while the between-subjects design misses it a large share of the time. Pairing is not a statistical nicety; it is the difference between a study that reliably answers the question and one that often comes back inconclusive. Whenever the same subjects can be measured under both conditions, pairing is usually the most powerful design available.
Step 10 of the chapter found big improvers, near-zero changers and two users who got slower. That split is worth taking seriously, because it may reveal where the redesign helps and where it does not. Here we characterize the groups by their speed on the old design, a first clue about who the non-responders are.
df['baseline'] = df['old_design_seconds'] # speed before the redesign (an average of old and new would contain the outcome)
df['group'] = np.select([df['diff'] < -5, df['diff'] > 10], ['responder', 'got slower'], 'little/no change')
print(df.groupby('group').agg(users=('diff','size'), mean_improvement=('diff','mean'), mean_baseline=('baseline','mean')).round(1))
fig, ax = plt.subplots(figsize=(7,3.3))
for grp, c in [('responder', EM), ('little/no change', GREY), ('got slower', RED)]:
s = df[df.group==grp]; ax.scatter(s.baseline, s['diff'], color=c, alpha=0.75, label=grp)
ax.axhline(0, color=INK, lw=1); ax.set(title='Improvement vs baseline speed', xlabel='old-design task time (s)', ylabel='new - old (s)'); ax.legend()
plt.tight_layout(); plt.show()
users mean_improvement mean_baseline group got slower 2 18.1 60.4 little/no change 12 0.6 61.4 responder 31 -14.9 63.1
Splitting at a five-second improvement, most users are clear responders while a minority barely move. Plotting each user's improvement against their speed on the old design is the start of an answer. Here the groups started at similar speeds (63.1 s for responders, 61.4 s for those with little change and 60.4 s for the two who got slower), so speed before the redesign does not explain who benefited. Grouping users by their own change also invites regression to the mean, and the split was chosen after seeing the data, so treat any pattern as a hypothesis to test in the next study.
The one idea to keep: when the same subjects can be measured under both conditions, pair them, because using each subject as their own control cancels the between-subject variance that otherwise hides the effect, and then match the test to the differences (t if normal, Wilcoxon or the sign test if not).