⚙️ Setup¶
import numpy as np
from scipy import stats
rng = np.random.default_rng(411)
print("ready")
ready
df = 9
tcrit = stats.t.ppf(0.975, df)
print(f"t critical (df=9) = {tcrit:.3f}")
print(f"normal z = 1.960")
t critical (df=9) = 2.262 normal z = 1.960
Answer: the critical value is t = 2.262, noticeably larger than z = 1.96. With only 9 degrees of freedom the t has heavier tails, so a small sample must clear a higher bar to reach significance, the price of estimating the spread from little data.
built = (rng.normal(0,1,size=(300_000,5))**2).sum(axis=1)
print(f"simulated mean of chi-square(5) = {built.mean():.3f} (theory = 5)")
simulated mean of chi-square(5) = 4.996 (theory = 5)
Answer: the simulated mean is about 5.0, matching the rule that a chi-square's mean equals its degrees of freedom. Each squared standard normal contributes a mean of 1, and five of them sum to 5.
xbar, s, n = 50, 8, 10
se = s/np.sqrt(n)
tcrit = stats.t.ppf(0.975, n-1)
lo, hi = xbar - tcrit*se, xbar + tcrit*se
print(f"SE = 8/sqrt(10) = {se:.3f}, t(df=9) = {tcrit:.3f}")
print(f"95% CI = 50 +/- {tcrit:.3f}*{se:.3f} = [{lo:.2f}, {hi:.2f}]")
SE = 8/sqrt(10) = 2.530, t(df=9) = 2.262 95% CI = 50 +/- 2.262*2.530 = [44.28, 55.72]
Answer: SE = 8/√10 ≈ 2.53, and with t(9) = 2.262 the interval is 50 ± 2.262(2.53) = [44.28, 55.72]. Using t instead of z makes the interval a bit wider, the honest cost of not knowing the true standard deviation.
observed = np.array([40, 55, 52, 53])
expected = np.full(4, 200/4)
chi2_stat, p = stats.chisquare(observed, expected)
print(f"chi-square = {chi2_stat:.3f}, df = 3, p-value = {p:.3f}")
print("-> consistent with equal colors" if p>0.05 else "-> evidence colors are NOT equal")
chi-square = 2.760, df = 3, p-value = 0.430 -> consistent with equal colors
Answer: the chi-square statistic is 2.76 with df = 3, giving p ≈ 0.430. Since p > 0.05, the data is consistent with equal colors; the deviations from 50 each are within normal sampling variation.
A = np.array([0.72,0.75,0.71,0.74,0.70,0.73,0.72,0.74])
B = np.array([0.75,0.78,0.74,0.77,0.74,0.76,0.75,0.78])
t_stat, p = stats.ttest_rel(B, A)
print(f"mean A = {A.mean():.3f}, mean B = {B.mean():.3f}")
print(f"paired t = {t_stat:.3f}, p-value = {p:.5f}")
print("-> B is significantly better" if p<0.05 else "-> not significant")
mean A = 0.726, mean B = 0.759 paired t = 19.858, p-value = 0.00000 -> B is significantly better
Answer: model B beats A on every fold, and the paired t-test gives a p-value far below 0.05, so the improvement is statistically significant. Pairing by fold removes the fold-to-fold difficulty and isolates the model effect, the standard way to compare algorithms fairly.