Survey results are where people most often misread statistics, quoting a rate as if it were exact, comparing averages of 1-to-5 ratings, or calling a weak association "strong." This case study handles all three traps on one survey, the careful way.
(1) What share would recommend, with a margin of error? (2) Does recommending depend on plan tier? (3) Do new and existing customers differ in satisfaction?
The Data & The Questions
One row per respondent, with a binary would_recommend, an ordinal
satisfaction_1_5, and categorical plan and segment. Three columns, three
different data types, three different tools.
One row per respondent with segment, plan,
region, satisfaction_1_5 (ordinal), and would_recommend (0/1).
The headline: about 64% would recommend. But this is a sample of customers, so 64% is an estimate, not a fact, it deserves a margin of error. And because a 1-to-5 rating is ordinal (the steps are not guaranteed equal), we will compare it with a rank-based test, not by averaging.
Three Questions, Three Tests
Each question maps to its data type. A rate gets an interval; two categoricals get a chi-square; an ordinal rating across groups gets a rank test.
| Question | Data type | Tool | H₀ / H₁ |
|---|---|---|---|
| Share who would recommend? | one proportion | estimate + margin of error | — (estimation) |
| Recommending vs plan? | two categorical variables | chi-square independence | independent / associated |
| Satisfaction: new vs existing? | ordinal (1-5), 2 groups | Mann-Whitney U | same distribution / they differ |
Why Mann-Whitney for satisfaction? A 1-to-5 scale is ordinal: we know 4 beats 3, but not that the gap equals the gap from 1 to 2, so averaging is shaky. Comparing ranks sidesteps that. For the chi-square we will confirm every expected cell count is at least ~5, and we will report Cramer's V so a small p from 720 respondents is not mistaken for a strong link.
The Analysis & Results
The margin of error frames the headline; chi-square and Cramer's V handle the plan link; Mann-Whitney compares the segments.
| Question | Result | Reading |
|---|---|---|
| Recommend rate | 64% ± 3.5 pts (95%), i.e. [61%, 68%] | healthy, quote as a range |
| Plan × recommend | χ² = 13.7, p ≈ 0.001, V ≈ 0.14 | real but modest association |
| Satisfaction (new vs existing) | means 3.58 vs 3.99, Mann-Whitney p ≈ 10⁻⁸ | existing clearly happier |
Three honest reads. The 64% recommend rate carries a ±3.5 point margin, so the truth is likely 61-68%. Recommending does depend on plan (p ≈ 0.001), but Cramer's V ≈ 0.14 says the link is modest, higher tiers recommend more, yet plan explains only a sliver of who recommends. And existing customers out-rate new ones on satisfaction (Mann-Whitney p ≈ 10⁻⁸) even though both medians are 4, the rank test catches a real shift in the whole distribution that the medians alone would hide.
The Statistician's Report
How to brief a head of customer experience without a stats lecture.
Read-out: a solid score, a clear lever, and an onboarding flag
What we found. About 64% of customers would recommend us, give or take 3.5 points (realistically 61-68%). Willingness to recommend rises with plan tier (a statistically clear pattern), and existing customers are more satisfied than new ones (a real, significant gap).
What it means. The 64% is healthy, but always quote it as a range, do not celebrate or panic over a one-to-two point move, that is within the noise. The plan link is real but modest: tier explains only a small part of who recommends, so "move people to Pro" is a lever, not a cure. The new-versus-existing gap is the actionable signal: newer customers are measurably less happy.
What to do. Track the recommend rate with its margin of error over time, and run a focused review of the new-customer onboarding experience, that is where the satisfaction gap is concentrated.
Caveats. Surveys suffer non-response and self-selection, the people who reply may not represent everyone, so treat these as directional. And satisfaction was compared by ranks, not averages, on purpose, because a 1-5 scale is ordinal.
This is the statistical backbone of any NPS or CSAT dashboard, and of feature-importance screening (chi-square between a categorical feature and an outcome). The discipline, margins of error on rates, effect sizes beside p-values, rank tests for ordinal scales, is what keeps a dashboard from over-claiming.
Work the survey in Python
The companion notebook explores the category counts and rating distribution, then uses statsmodels (proportion_confint) for the margin of error and scipy for the chi-square, Cramer's V, and the Mann-Whitney rank test, with plots of recommend-by-plan and satisfaction.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, statsmodels, and openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓A survey rate is an estimate: 64% ± 3.5 pts, always report the margin of error.
- ✓Two categoricals (plan × recommend) → chi-square, with Cramer's V for strength (here only ~0.14, modest).
- ✓Ordinal ratings (1-5) → Mann-Whitney on ranks, not a t-test on averages.
- ✓Result: significant plan link (modest) and existing > new on satisfaction (p ≈ 10⁻⁸), despite equal medians.
- ✓Action: track the rate with its interval; review new-customer onboarding; mind non-response bias.
Quiz: Test Yourself
Eight quick questions on this case study. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
The finale brings it home to the factory floor. Manufacturing Quality combines a one-sample test against spec, an ANOVA across lines, and a defect-by-shift chi-square, and closes Inference Case Studies.