A pollster reports "64% approve, and that beats the 60% benchmark." A product team says "B converts better than A." Both are z-tests: when n is large or we are dealing with proportions, the standardized statistic follows a standard normal curve, and the whole test reduces to "how many standard errors from the null?"
Standardize the estimate into a z, read the tail area off the normal curve, and decide. For proportions the standard error comes from the null (one sample) or a pooled rate (two samples).
When z, Not t
The z-test and the t-test answer the same question; they differ only in the reference curve. Use z when the population standard deviation is known, when the sample is large enough that s is essentially σ, or for proportions. Use t (next chapter) when σ is unknown and the sample is small.
In the notebook a large-sample z-test (n = 400) gives a p-value indistinguishable from the t-test, the curves have merged. That is why z is the natural default for big samples, and the only sensible choice for proportions, where the variance is pinned down by the proportion itself rather than estimated separately.
The One-Proportion z-Test
For a yes/no outcome we test H₀: p = p₀. The key move: under the null the standard error is a known number, √(p₀(1−p₀)/n), because the null value fixes the variance. The statistic z = (p̂ − p₀) / SE₀ is then standard normal.
In the notebook, 545 successes in 1,000 trials is 2.85 standard errors above 0.5 (p ≈ 0.004), so the rate genuinely differs from one-half. If the success-failure condition fails (too few of either outcome), the normal approximation breaks down and an exact binomial test is the right fallback.
The Two-Proportion z-Test
To compare two groups we test H₀: p₀ = p₊ (equivalently, the difference is 0). Under that null both groups share one true rate, so we estimate it with the pooled proportion and use it in the standard error.
The standard error becomes √(p̂(1−p̂)(1/n₀ + 1/n₊)), and z is the difference divided by that. This single test underlies comparing two conversion rates, two pass rates, or two approval numbers, and it is exactly what an A/B test runs on its primary metric (see A/B Testing & Online Experiments).
Real-World Example: An Approval Poll
A poll of 820 respondents records whether each approves of a policy. The campaign claims approval is above 60%. We answer two questions: does overall approval really exceed the 60% benchmark (one-proportion z), and does approval differ between the youngest and oldest age bands (two-proportion z)?
One row per respondent with a binary approve, plus region
and age_group.
| Test | Estimate | z | p-value | Verdict |
|---|---|---|---|---|
| Approval vs 60% (one-prop) | p̂ = 64.3% | 2.49 | ≈ 0.013 | reject H₀: higher than 60% |
| Young vs old (two-prop) | 70.7% vs 56.8% | 3.22 | ≈ 0.001 | reject H₀: a real age gap |
Overall approval (64.3%) sits about 2.5 standard errors above the 60% benchmark, so the one-proportion test rejects the claim, approval is genuinely higher. The age contrast is even stronger: 70.7% of the youngest band versus 56.8% of the oldest gives z ≈ 3.22 (p ≈ 0.001), a clear, real difference rather than sampling noise. With 820 respondents and both outcomes well represented, the success-failure condition holds and the normal approximation is trustworthy.
z-Tests in Machine Learning & AI
Proportion z-tests are the default for the rate-based metrics that dominate applied ML.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| One-proportion z | "Does accuracy beat a target?" | is click-through above the 5% goal? |
| Two-proportion z | Comparing two models' success rates | conversion of model A vs model B |
| Large-sample z for a mean | Big-data mean comparisons | average latency this week vs last |
| Pooled standard error | The core of A/B significance | online experiment on a primary metric |
Most production metrics are rates, click-through, conversion, error, churn, and the two-proportion z-test is the standard way to decide whether a new model or feature moved one of them. Because web-scale samples are huge, the normal approximation is essentially exact, so the z-test (and the matching confidence interval on the difference) is what powers experimentation platforms. The same caution from the Significance, p-values & Errors chapter applies: with millions of users a microscopic lift is "significant", so report the size of the effect, not just the z.
Run every z-test in Python
The companion notebook builds the large-sample one-sample z, the one-proportion z (with the success-failure
check), and the two-proportion z with a pooled standard error, then loads
z-tests--approval_poll.xlsx to test approval against 60% and compare age groups.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓z-test: standardize to z = (estimate − null)/SE and read the tail off the standard normal curve.
- ✓Use z for large samples (s ≈ σ) and for proportions, where the null fixes the standard error.
- ✓One-proportion SE uses p₀; two-proportion SE uses a pooled rate, valid under the success-failure condition.
- ✓Real data: approval beats 60% (z = 2.49, p ≈ 0.013) and differs by age (z = 3.22, p ≈ 0.001).
- ✓In ML/AI: rate metrics (CTR, conversion, error) live on the two-proportion z, the heart of A/B testing.
Practice Challenges
Five short challenges, beginner to intermediate. Try them before checking the solutions.
One-proportion z-test
In 1,000 trials you see 470 successes. Test H₀: p = 0.5 (two-sided).
Two-proportion z-test
Group A: 90/500. Group B: 130/520. Test H₀: p₀ = p₊.
Large-sample z for a mean
A sample of 500 has mean 51 and sd 10. Test H₀: μ = 50 (two-sided).
Check the validity condition
For n = 40 and p₀ = 0.05, is the normal approximation valid? Use the success-failure rule.
Real data: approval poll
Load z-tests--approval_poll.xlsx and run the one-proportion z-test of H₀: p = 0.60.
approve.mean().A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on z-tests. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
The z-test assumes a known σ or a large sample. t-Tests handles the common reality, an unknown σ and a modest sample, with the one-sample, two-sample, and paired t-tests.