Chapter 75 · Solutions
Significance & Errors — Worked Solutions ✅
Five challenges, each verified in code.
⚙️ Setup¶
In [1]:
import numpy as np
from scipy import stats
import pandas as pd
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
rng=np.random.default_rng(750)
CHALLENGE 1
Verify the false-positive rate
Run 10,000 two-sample t-tests on pure noise (both groups N(0,1), n=30) and confirm ~5% are 'significant'.
In [2]:
sig=np.mean([stats.ttest_ind(rng.normal(0,1,30),rng.normal(0,1,30)).pvalue<0.05 for _ in range(10000)])
print(f"false-positive rate = {sig:.3f} (target 0.05)")
false-positive rate = 0.050 (target 0.05)
CHALLENGE 2
Estimate power
For a true mean of 0.4 (sd 1), n=50, estimate the power of a one-sample t-test at alpha=0.05.
In [3]:
pw=np.mean([stats.ttest_1samp(rng.normal(0.4,1,50),0).pvalue<0.05 for _ in range(5000)])
print(f"estimated power = {pw:.2f}")
estimated power = 0.80
CHALLENGE 3
Power vs sample size
Show power rising with n (effect 0.4) for n in [20,50,100,200].
In [4]:
for n in [20,50,100,200]:
pw=np.mean([stats.ttest_1samp(rng.normal(0.4,1,n),0).pvalue<0.05 for _ in range(3000)])
print(f"n={n:>3}: power={pw:.2f}")
n= 20: power=0.40
n= 50: power=0.80
n=100: power=0.98
n=200: power=1.00
CHALLENGE 4
Significance is not importance
With n=50,000 and a true effect of 0.01, show the test is 'significant' yet the effect size is negligible.
In [5]:
x=rng.normal(0.01,1,50000); p=stats.ttest_1samp(x,0).pvalue
print(f"p={p:.3e} (significant) but d={x.mean()/x.std():.3f} (negligible)")
p=1.606e-02 (significant) but d=0.011 (negligible)
CHALLENGE 5
Real data: power of the battery test
Load significance-p-values-and-errors--battery_life.xlsx, run the one-sided t-test of H0: mu=10, and report the power at the observed effect.
In [6]:
try: bat = pd.read_excel("../../data/significance-p-values-and-errors--battery_life.xlsx", sheet_name="Batteries")
except FileNotFoundError: bat = pd.read_excel(BASE+"significance-p-values-and-errors--battery_life.xlsx", sheet_name="Batteries")
from scipy.stats import nct, t as tdist
x=bat.life_hours; n=len(x); d=(x.mean()-10)/x.std(ddof=1)
res=stats.ttest_1samp(x,10)
power=nct.cdf(tdist.ppf(0.05,n-1), n-1, d*np.sqrt(n))
print(f"one-sided p={res.pvalue/2:.4f}, d={d:.2f}, power={power:.2f}")
one-sided p=0.0481, d=-0.24, power=0.51
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher