You have met point estimates and confidence intervals, the logic of hypothesis testing, z-, t-, F-, chi-square, and rank-based tests. The skill now is orchestration: turning a messy real-world question into the right test, run correctly, and reported honestly. That is what the next five chapters practice, and this one is the map.
Estimation and testing are two views of the same data; four questions pick the test; assumptions come first; and a result is only finished when it carries an effect size and an interval, not just a p.
The Two Sides of Inference
Estimation asks "what is the value, and how sure are we?", a point estimate wrapped in a confidence interval. Testing asks "is the effect real, or could chance explain it?", H₀ vs H₁ with a p-value. They are mirror images of one another.
In the notebook, the same change scores give a 95% CI of about [1.1, 5.3] (which excludes 0) and a test with p ≈ 0.003 (which rejects), the two always agree. Use both in tandem: the test says whether there is an effect, the interval says how large it plausibly is.
The Master Decision Map
Answer the four questions and the test is almost forced. This is the same map from the Choosing the Right Test chapter, widened to include estimation and proportions, the version the case studies will use out loud.
| Outcome | Groups | Design | Test (rank-based fallback) |
|---|---|---|---|
| Numeric | 1 vs a value | — | one-sample t-test (Wilcoxon) |
| Numeric | 2 | paired | paired t-test (Wilcoxon signed-rank) |
| Numeric | 2 | independent | two-sample Welch t (Mann-Whitney U) |
| Numeric | 3+ | independent | one-way ANOVA + Tukey (Kruskal-Wallis) |
| Proportion | 1 or 2 | — | one- / two-proportion z-test |
| Categorical | 2 variables | — | chi-square test of independence |
Assumption Checklists
The fourth question is where most analyses go wrong. Check before you choose, then switch to the rank-based twin when the assumption fails.
| Test family | Check | If it fails |
|---|---|---|
| t-test / ANOVA | roughly normal, or n ≥ ~30 (CLT); no wild outliers | Mann-Whitney / Kruskal-Wallis |
| Welch vs pooled t | unequal variances? prefer Welch by default | Welch handles it |
| Chi-square | expected count ≥ ~5 in every cell | Fisher's exact test |
| Proportion z | at least ~10 successes and ~10 failures | exact binomial test |
| Paired tests | data are genuinely paired (same units twice) | do not treat as independent |
The notebook shows the quick programmatic checks: skewness and the Shapiro-Wilk test for normality, and the minimum expected count for a chi-square. The discipline is simple, check, then choose, never the reverse.
Reading a Result Honestly
A p-value is half an answer. With a large enough sample, a trivial effect is "significant"; with too small a sample, a real effect hides. The cure is to always report three things together.
| Report | Answers |
|---|---|
| p-value | could chance alone explain this? (yes/no at α) |
| Effect size | how big is it? (Cohen's d, lift, Cramer's V, η²) |
| Confidence interval | how precisely do we know it? |
In the notebook, a tiny true effect of 0.06 at n = 50,000 gives p ≈ 10⁻⁴¹ (wildly "significant") with an effect size of 0.06 (negligible), the textbook reminder that significance is not importance. And "not significant" never means "no effect", it may just mean the study lacked power. Every case study ahead closes with this honest, three-part read.
The Five Case Studies Ahead
Each of the next five chapters takes one real, downloadable dataset from question to plain-English recommendation, narrating every decision: the hypotheses, the test choice, the assumption checks, the numbers, and a statistician's report written for a non-statistician.
Web A/B TestThe decision rule as runnable code
The companion notebook shows estimation and testing agreeing on the same data via statsmodels' DescrStatsW, implements a choose_test() helper for the whole decision map, runs the assumption checks (skewness, Shapiro-Wilk, expected counts), and demonstrates why every p-value needs an effect size beside it.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Estimation and testing are two views of one dataset; a CI that excludes 0 matches a test that rejects H₀.
- ✓Four questions (outcome, groups, design, assumptions) pick the test almost uniquely.
- ✓Check assumptions first: normality / n, expected counts, pairing; switch to a rank-based test when they fail.
- ✓Report three things: a p-value, an effect size, and an interval, significance is not importance.
- ✓Next: five case studies apply this playbook end to end, each with a statistician's report.
Quiz: Test Yourself
Eight quick questions on the inference playbook. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
With the playbook in hand, Case Study: A Web A/B Test takes a real experiment from question to ship decision, framing the hypotheses, checking power, running the two-proportion test, and reporting the lift with an interval.