You ran a test and got p = 0.03. You declare victory. But what did you actually promise, and how often will that promise betray you? Understanding significance, the two error types, and power is what separates careful inference from p-value theater.
α controls false alarms; power controls misses. A p-value is the chance of data this extreme if H₀ is true, never the probability that H₀ is true, and never a measure of how big or important the effect is.
What "Statistically Significant" Means
Calling a result significant at α = 0.05 is a statement about a procedure, not a single study: over the long run, if the null is true, this rule rejects it only 5% of the time. That 5% is a budget for false alarms you decide to tolerate, before seeing the data.
The notebook confirms it: across 20,000 tests where H₀ is true, exactly 5.0% come out "significant", and the p-values are perfectly uniform. Those rejections are not discoveries, they are the false-alarm tax you knowingly pay. Lowering α (say to 0.01) buys fewer false alarms, at the cost of more misses.
Two Ways to Be Wrong, and Power
Reality is binary (an effect exists or it does not) and so is our verdict (reject or not), giving four outcomes. Two are correct; two are errors. A Type I error rejects a true null (false alarm); a Type II error keeps a false null (a miss).
| Reality ↓ / Verdict → | Reject H₀ | Fail to reject H₀ |
|---|---|---|
| H₀ true (no effect) | Type I error — rate α | correct (true negative) |
| H₀ false (real effect) | correct — power = 1 − β | Type II error — rate β |
In the notebook, a true effect of 0.5 is caught only 30% of the time at n = 10 but 99% at n = 80, power rises with sample size and with effect size. The takeaway: "no significant difference" from a small, underpowered study is nearly uninformative. You may simply have lacked the power to see a real effect.
What a p-value Is Not
Most p-value disasters come from three misreadings. Naming them is the cure.
| Myth | Reality |
|---|---|
| "p is the probability H₀ is true" | p is P(data this extreme | H₀ true), a statement about the data, not the hypothesis |
| "a small p means a big effect" | with enough data, a trivial effect gets a tiny p, report an effect size too |
| "one significant result among many counts" | test 20 noise variables and ~1 is "significant" by luck, correct for multiplicity |
The fix for multiplicity is a correction (Bonferroni divides α by the number of tests; false-discovery-rate control is gentler). The fix for "significant but tiny" is to always report an effect size and a confidence interval. In the notebook a true effect of 0.02 at n = 100,000 gives p ≈ 10⁻¹⁰, overwhelmingly "significant" and utterly unimportant.
Real-World Example: A Battery That Barely Fails
A battery is rated at 10.0 hours. An engineer tests 50 units and suspects they run short. The test is one-sided: H₀: μ = 10 vs H₁: μ < 10. The verdict is "significant", but the more important question is whether a test this size could be trusted to find a real shortfall at all.
One row per tested unit with life_hours, plus the model and
factory.
| Quantity | Value | Reading |
|---|---|---|
| Sample mean | 9.82 h (sd 0.77, n = 50) | a touch under 10 |
| t statistic | −1.70 (df = 49) | just past the cutoff |
| One-sided p-value | ≈ 0.048 | barely < 0.05: reject H₀ |
| Effect size (d) | ≈ −0.24 | small |
| Power at this effect | ≈ 0.51 | would miss it ~half the time |
| n for 90% power | ≈ 150 units | need a bigger sample |
This is a textbook fragile result. The p-value (≈ 0.048) clears 0.05 by a whisker, but the power is only about 0.51, meaning a test of this size would fail to detect a shortfall this small nearly half the time. The honest report is not "the battery fails" but "the data suggest a small shortfall (d ≈ −0.24); a confirmatory test with about 150 units is needed to be sure." A low p with low power is a prompt for more data, not a press release.
Significance & Power in Machine Learning & AI
Error rates and power are everywhere in ML evaluation, from leaderboard claims to monitoring dashboards.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Type I error (α) | False-positive alerts | a drift monitor crying wolf |
| Type II error (β) | Missed regressions | shipping a model that quietly got worse |
| Power / sample size | Experiment design | how many users an A/B test needs |
| Multiple comparisons | Many-metric dashboards | 20 metrics → ~1 "significant" by luck |
Benchmark leaderboards are multiple-comparison machines: with dozens of models and tasks, some gaps are significant by chance, which is why reproducibility and held-out test sets matter so much. Production ML adds the cost asymmetry, a Type II error (failing to notice a model has degraded) is often far more expensive than a Type I (a false drift alert), so teams set α and target power deliberately. And every A/B test needs a power analysis up front: launching with too few users guarantees you will miss the very improvements you are hunting for.
See significance, errors, and power in code
The companion notebook verifies the 5% false-alarm rate, simulates power as a function of sample size and
effect size, demonstrates the multiple-comparisons and significance-vs-importance traps, and loads
significance-p-values-and-errors--battery_life.xlsx to test the 10-hour claim and compute its power.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓α is the false-alarm (Type I) rate you accept; under H₀, p-values are uniform, so 5% are "significant" by chance.
- ✓Type II error (β) is missing a real effect; power = 1 − β rises with sample size and effect size.
- ✓A p-value is not P(H₀ true), not an effect size, and not safe to cherry-pick from many tests.
- ✓Real data: the battery test is barely significant (p ≈ 0.048) with only ~51% power, fragile evidence needing more units.
- ✓In ML/AI: balance false alerts vs missed regressions, correct for many-metric dashboards, and power your A/B tests.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
Verify the false-positive rate
Run 10,000 two-sample t-tests on pure noise (both groups N(0,1), n = 30) and confirm ~5% are significant.
pvalue < 0.05 over many simulated pairs.Estimate power
For a true mean of 0.4 (sd 1) and n = 50, estimate the power of a one-sample t-test at α = 0.05.
Power vs sample size
Show power rising with n (effect 0.4) for n in {20, 50, 100, 200}.
Significance is not importance
With n = 50,000 and a true effect of 0.01, show the test is significant yet the effect size is negligible.
mean/std.Real data: power of the battery test
Load significance-p-values-and-errors--battery_life.xlsx, run the one-sided t-test of H₀: μ = 10, and report the power at the observed effect.
nct.cdf(t.ppf(0.05, n−1), n−1, d√n).A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on significance, errors, and power. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now read a p-value honestly and weigh both error types. z-Tests puts the machinery to work on means and proportions when the sample is large, the workhorse tests behind polls and conversion rates.