Ask a beginner how good their model is and they say "95% accurate." Ask a professional and they ask "accurate at what, and compared to what?" The single number can hide total failure, so real evaluation uses a small family of metrics, each answering a different question.
Never trust accuracy alone on imbalanced data: read the confusion matrix, weigh precision against recall by the cost of each error, use ROC-AUC or average precision to compare models, and for regression report RMSE (or MAE) plus R-squared.
Why Accuracy Lies
Accuracy is the fraction of predictions that are correct. It is fine when the classes are balanced, and dangerously misleading when they are not, which is the case for most problems worth solving: fraud, disease, defects, and churn are all rare.
The notebook's fraud data is only 13% fraud. A lazy model that predicts "legitimate" for every transaction is 87% accurate, and catches zero fraud, a recall of 0. Our real model scores almost the identical accuracy (87.3%), yet it is genuinely useful because it actually flags fraud. Accuracy cannot tell the two apart, because the 87% majority swamps it. To see what a model does on the cases that matter, we have to look at the minority class directly.
The Confusion Matrix: Precision, Recall, F1
Every classification metric is built from four counts, the confusion matrix, which cross-tabulates what the model predicted against the truth.
Precision asks: of the transactions we flagged, how many were really fraud? (it punishes false alarms). Recall asks: of all the fraud, how much did we catch? (it punishes misses). On the fraud model they are 0.51 and 0.31, very different from the flattering 87% accuracy. F1, their harmonic mean, gives one balanced score. Which you optimize is a business decision: a fraud team dreads missed fraud (recall), an email filter dreads blocking real mail (precision).
The Threshold, PR Curve & ROC-AUC
A classifier really outputs a probability; a threshold turns it into yes/no. The default 0.5 is rarely optimal. Lower it and you flag more, so recall rises and precision falls (in the notebook, moving from 0.5 to 0.25 lifts recall from 0.31 to 0.43 while precision drops from 0.51 to 0.38). The precision-recall curve traces every operating point, and you pick the threshold from the cost of a miss versus a false alarm.
The notebook makes "by cost" concrete instead of hand-wavy: charge $500 for a missed fraud and $10 to review a false alarm, then sweep every threshold and add up the bill. Expected cost bottoms out near a threshold of 0.02 (about $2,900 on the test set), roughly 7× cheaper than blindly using 0.5 (about $20,170). The optimal threshold is simply the one that minimizes expected cost, a business calculation, not a statistical default.
To compare models independently of any threshold, use the ROC curve and its area, AUC.
The ROC curve plots the true-positive rate (recall) against the false-positive rate as the threshold sweeps from strict to loose; the AUC is the area beneath it, a single threshold-free number equal to the probability the model ranks a random fraud above a random legit transaction. 0.5 is a coin flip, 1.0 is perfect; our model scores about 0.81. AUC is ideal for comparing models, but when positives are very rare the precision-recall / average-precision view often reflects real usefulness better.
Real-World Example: Scoring the Fraud Model
Put the whole toolkit on one table, and the difference between "looks great" and "actually works" becomes obvious.
One row per transaction with amount, hour,
account_age_days, txns_last_hour, foreign, and
high_risk_merchant, plus the imbalanced label is_fraud (about 13% fraud).
| Model / setting | Accuracy | Precision | Recall | AUC |
|---|---|---|---|---|
| Flag nothing (baseline) | 0.871 | — | 0.00 | 0.50 |
| Gradient boosting @ 0.5 | 0.873 | 0.51 | 0.31 | 0.81 |
| Gradient boosting @ 0.25 | lower | 0.38 | 0.43 | 0.81 |
The baseline and the model have the same accuracy, yet one is worthless and the other catches a third of the fraud, exactly what accuracy conceals. And when the target is a number instead of a class, the parallel metrics take over: on a simple regression the notebook reports MAE = 2.29 and RMSE = 2.90 (the typical error, in the target's units, with RMSE punishing big misses harder) and R-squared = 0.85 (the share of variance explained). Report an error metric and R-squared: one gives the size of the typical miss, the other the proportion explained.
Evaluation in Machine Learning & AI
Choosing the metric is not a formality, it defines what "good" means, and different problems demand different measures.
| Situation | Reach for |
|---|---|
| Balanced classes | accuracy, F1 |
| Imbalanced classes (fraud, disease) | precision, recall, F1, average precision (PR-AUC) |
| Comparing / ranking models | ROC-AUC |
| Regression | RMSE or MAE, plus R-squared |
| Probabilities must be trustworthy | log loss, calibration curves |
The metric is the product decision. A cancer-screening model is tuned for recall (never miss a case, tolerate false alarms); a spam filter is tuned for precision (never trash real mail). Picking the wrong one, most infamously reporting accuracy on imbalanced data, is a leading way that models look great in a demo and fail in production. At the frontier the same logic scales into whole benchmark suites: large language models are judged by families of task-specific metrics, and a deep, active field of evaluation research exists precisely because "how good is it?" is never a single number. Learning to choose and read the right metric is one of the most valuable skills in applied ML.
Evaluate a model in Python
The companion notebook exposes how accuracy lies against a do-nothing baseline, draws the confusion matrix, computes precision / recall / F1, sweeps the threshold with a precision-recall curve, plots the ROC curve and AUC for two models, and reports the regression metrics MAE / MSE / RMSE / R-squared, every cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓Accuracy lies on imbalanced data: a do-nothing model scored 87% while catching zero fraud.
- ✓Precision (of flagged, how many real) and recall (of real, how many caught) come from the confusion matrix; F1 balances them.
- ✓The threshold is a knob: lower it for more recall, raise it for more precision; pick it from the cost of each error.
- ✓ROC-AUC scores ranking regardless of threshold (0.5 random, 1 perfect); prefer average precision when positives are rare.
- ✓For regression: RMSE or MAE for the typical error size, plus R-squared for the share of variance explained.
Practice Challenges
Five short challenges on the fraud table. Try them with scikit-learn before checking the solutions.
Accuracy vs a do-nothing baseline
Compare a most-frequent DummyClassifier to the real model on accuracy and recall.
DummyClassifier(strategy="most_frequent").Precision, recall, F1
From the confusion matrix, report the three classification scores.
precision_score, recall_score, f1_score.The threshold trade-off
Show recall rises and precision falls as the decision threshold is lowered.
predict_proba, then compare at 0.5, 0.3, 0.2.ROC-AUC comparison
Compare the AUC of gradient boosting and logistic regression.
roc_auc_score(y, model.predict_proba(X)[:,1]).Regression metrics
On a simple regression, report MAE, RMSE, and R-squared.
mean_absolute_error, mean_squared_error, r2_score.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on model evaluation. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now measure a model honestly, but good numbers hide traps. Model Pitfalls covers what goes wrong, overfitting and the bias-variance trade-off, imbalanced data and SMOTE, and the cross-validation strategies that keep your estimates trustworthy.