Choose the right statistical test in four questions
Why it matters
Results from pilots, A/B tests and model comparisons often reach a decision as a single p-value, and if the wrong test produced it, even a perfect calculation gives a meaningless answer.1 Ask which test was used, and ask for the size of the effect and its confidence interval beside the p-value, because with enough data a trivial effect is significant.2
Four questions about the data almost force the choice of test, and the discipline lies in asking them before running anything.1,2 A classic slip is treating paired measurements as independent, as when two models are scored on the same cross-validation folds, and it can erase a real improvement or manufacture a false one.1
What you will do
The lab runs on a clinic dataset written for the book, with 200 patients, their blood pressure before and after treatment, their wait times at three clinics and whether they were satisfied.1
- Ask the four questions in order. Is the outcome numeric or categorical, how many groups are there, are the groups independent or paired, and do the test’s assumptions hold?1
- Read the test off the decision map. Switch to its rank-based twin when the data are skewed or have outliers, and to Fisher’s exact test when a cell’s expected count falls below about 5.1
- Route four questions about the clinic through the map. Blood pressure fell by about 9 mmHg within patients (a paired t-test), and the new treatment lowered it more than the standard one did, by 11.6 mmHg against 6.3 (a Welch t-test).1
- Report the other two answers with the same care, although neither is significant. Satisfaction shows no clear link to treatment (a p-value of about 0.27), and wait times do not differ by clinic under ANOVA or under its rank-based twin.1
- Close each result with its p-value, effect size and confidence interval, as the review chapter does for the five case studies that follow it.2
Before you start
All the notebook needs is a free Google account for Colab, which already has SciPy and statsmodels. The notebook fetches the clinic data from the book’s repository, so it runs free with no setup. It helps to have met the t-test and the chi-square test, and the review chapter turns the whole map into a choose_test() helper with the assumption checks in code.
Articles it puts into practice
Every AI claim is a statistical claim sets out three questions to ask of any AI number. The right test is how an analyst answers two of them, “Compared with what?” and “How sure?”
Found this useful? Follow me on LinkedIn for the next one.
Working on this? Connect and tell me what you are seeing.