Chapter 111 · Solutions
Model Evaluation · Solutions
Five challenges, each verified in code.
Chapter 111 · Practice Challenges, Solved¶
Five exercises on model evaluation, worked in scikit-learn on the Chapter 111 fraud table.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.preprocessing import StandardScaler; from sklearn.pipeline import make_pipeline
from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
roc_auc_score, mean_absolute_error, mean_squared_error, r2_score, confusion_matrix)
try: df = pd.read_excel('../../data/model-evaluation--transactions.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'model-evaluation--transactions.xlsx', sheet_name='Data')
feat = ['amount','hour','account_age_days','txns_last_hour','foreign','high_risk_merchant']
X, y = df[feat], df['is_fraud']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
clf = GradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
print('loaded', df.shape, '| fraud rate', round(y.mean(),3))
loaded (1500, 8) | fraud rate 0.129
CHALLENGE 1
Accuracy vs a do-nothing baseline
Compare a most-frequent DummyClassifier to the real model on accuracy and recall.
In [3]:
d = DummyClassifier(strategy='most_frequent').fit(X_tr, y_tr)
print(f'dummy : acc {accuracy_score(y_te, d.predict(X_te)):.3f} recall {recall_score(y_te, d.predict(X_te)):.3f}')
print(f'model : acc {accuracy_score(y_te, clf.predict(X_te)):.3f} recall {recall_score(y_te, clf.predict(X_te)):.3f}')
dummy : acc 0.871 recall 0.000 model : acc 0.873 recall 0.310
Solution. The dummy matches the model on accuracy but has zero recall, proof that accuracy alone is blind on imbalanced data.
CHALLENGE 2
Precision, recall, F1
From the confusion matrix, report the three classification scores.
In [4]:
p = clf.predict(X_te)
print('confusion matrix:\n', confusion_matrix(y_te, p))
print(f'precision {precision_score(y_te,p):.3f} recall {recall_score(y_te,p):.3f} F1 {f1_score(y_te,p):.3f}')
confusion matrix: [[375 17] [ 40 18]] precision 0.514 recall 0.310 F1 0.387
Solution. Precision = of flagged, how many were fraud; recall = of all fraud, how many caught; F1 balances them.
CHALLENGE 3
The threshold trade-off
Show recall rises and precision falls as the decision threshold is lowered.
In [5]:
pr = clf.predict_proba(X_te)[:,1]
for t in [0.5, 0.3, 0.2]:
pp=(pr>=t).astype(int); print(f'threshold {t}: precision {precision_score(y_te,pp):.2f}, recall {recall_score(y_te,pp):.2f}')
threshold 0.5: precision 0.51, recall 0.31 threshold 0.3: precision 0.38, recall 0.40 threshold 0.2: precision 0.34, recall 0.48
Solution. Lowering the threshold flags more transactions, catching more fraud (recall up) at the cost of more false alarms (precision down).
CHALLENGE 4
ROC-AUC comparison
Compare the AUC of gradient boosting and logistic regression.
In [6]:
logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)
print(f'GBM AUC = {roc_auc_score(y_te, clf.predict_proba(X_te)[:,1]):.3f}')
print(f'logit AUC = {roc_auc_score(y_te, logit.predict_proba(X_te)[:,1]):.3f}')
GBM AUC = 0.810 logit AUC = 0.820
Solution. AUC is threshold-free, the probability the model ranks a random fraud above a random legit transaction, ideal for comparing models.
CHALLENGE 5
Regression metrics
On a simple regression, report MAE, RMSE, and R-squared.
In [7]:
rng=np.random.default_rng(5); x=rng.uniform(0,10,200); t=2.5*x+5+rng.normal(0,3,200)
r=LinearRegression().fit(x.reshape(-1,1),t); yh=r.predict(x.reshape(-1,1))
print(f'MAE {mean_absolute_error(t,yh):.2f} RMSE {mean_squared_error(t,yh)**0.5:.2f} R2 {r2_score(t,yh):.3f}')
MAE 2.29 RMSE 2.90 R2 0.852
Solution. MAE and RMSE give the typical error size in the target's units (RMSE punishes big misses more); R-squared gives the share of variance explained.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher