88.7% accurate and missing almost every case
A model built to predict depression from seven survey measures reached 88.7% accuracy on 134,897 adults. At its default setting it flagged 805 of them, although about one in nine had reported a depressive episode, so nearly all of its accuracy came from answering “no”.1 Accuracy alone could not tell that model apart from a useful one.
Accuracy rewards the majority answer
In the analysis sample, 119,664 adults reported no depressive episode and 15,233 did, so a model that always answers “no” is right 88.7% of the time.1 The penalised logistic regression in the analysis matched that score at the default cut-off of 0.50, and its Cohen’s kappa was 0.04, effectively zero.1 Kappa measures agreement beyond what chance alone would produce, which is why it exposed what the accuracy figure hid. A kappa near zero means the model agreed with the truth about as often as chance would, given how rare the outcome is.
The model itself carried real signal. Its AUC of 0.7406 means it ranked a randomly chosen person with depression above a randomly chosen person without it about three times in four.1 The failure was in the decision rule, because the default cut-off of 0.50 sat far above the probabilities the model gives for an outcome this rare. AUC judges the ranking across every possible cut-off, and accuracy judges a single one, which is how the two can disagree so sharply.
| Cut-off | Depressed adults found | Precision | Kappa | Adults flagged |
|---|---|---|---|---|
| 0.50 | 2.7% | 50.8% | 0.040 | 805 (1%) |
| 0.30 | 15.8% | 37.5% | 0.167 | 6,421 (5%) |
| 0.20 | 39.0% | 29.0% | 0.233 | 20,496 (15%) |
| 0.15 | 54.2% | 24.2% | 0.212 | 34,070 (25%) |
| 0.10 | 73.5% | 19.6% | 0.160 | 56,950 (42%) |
Lowering the cut-off to 0.20 finds 39% of the people with depression, at the cost of flagging 15% of everyone for a closer look.1 At that cut-off, 29% of the people flagged have depression, so about seven in ten flags are false alarms.1 That rate suits a quick follow-up question and would be far too high for anything costly or intrusive. Which row is right depends on what happens to a person the model flags, and a cheap follow-up justifies a lower threshold than a costly intervention does.
The outcome is rare enough that people without depression outnumber those with it by 7.86 to 1. Class weighting and resampling, the other standard responses to imbalance, move the model along the same trade-off without adding information, so a team still has to choose where on the curve to sit.1
The Visual Handbook’s chapter on model evaluation shows the same pattern with fraud, where a model that labels every transaction legitimate is 87% accurate and catches no fraud at all.2 Any outcome that is rare enough produces the same illusion, from fraud and equipment failure to an agent’s request that needs a human to review it.
What to do next
- CIO. Ask for the base rate with every accuracy figure, and for the share of the cases that matter that the model actually finds.
- Analyst. Report recall, precision and kappa beside accuracy for any classifier whose positive class is rare.
- Data Scientist. Set the cut-off from the cost of each kind of error, and show the people who will use the model a table like the one above before they choose.
Go deeper
The survey analysis shows the classifier, every cut-off and the code behind them. Chapter 115 of the Visual Handbook covers the confusion matrix, precision and recall, the threshold and the precision-recall curve, with a notebook to run.
Try it
Catch a model that is 88.7% accurate and misses almost every case works through the same checks on card-fraud data, in a visual guide and a notebook that runs in Colab.
Found this useful? Follow me on LinkedIn for the next one.
Working on this? Connect and tell me what you are seeing.
Read chapter 115 of the Visual Handbook and run its notebook yourself.