Chapter 109 · Solutions
The Machine Learning Workflow · Solutions
Five challenges, each verified in code.
Chapter 109 · Practice Challenges, Solved¶
Five exercises on the ML workflow, worked in scikit-learn on the Chapter 109 loan-default table.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.tree import DecisionTreeClassifier
try: df = pd.read_excel('../../data/the-machine-learning-workflow--loans.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'the-machine-learning-workflow--loans.xlsx', sheet_name='Data')
feat = ['income_k','loan_amount_k','term_months','credit_score','age','employment_years','debt_to_income','num_open_accounts','prior_defaults','late_payments_12m']
X, y = df[feat], df['default']
print('loaded', df.shape, '| default rate', round(y.mean(),3))
loaded (1000, 13) | default rate 0.592
CHALLENGE 1
Split first
Hold out 30% as a stratified test set and confirm the default rate matches in both parts.
In [3]:
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.30, random_state=1, stratify=y)
print(f'train {len(X_tr)} rows (default {y_tr.mean():.1%}) | test {len(X_te)} rows (default {y_te.mean():.1%})')
train 700 rows (default 59.1%) | test 300 rows (default 59.3%)
Solution. stratify=y keeps the class balance nearly identical (59.1% against 59.3%), so the test set is a fair mirror of the training set.
CHALLENGE 2
The overfitting gap
Grow a full-depth tree and compare its training accuracy to its cross-validated accuracy.
In [4]:
tree = DecisionTreeClassifier(random_state=0).fit(X_tr, y_tr)
train_acc = tree.score(X_tr, y_tr)
cv_acc = cross_val_score(DecisionTreeClassifier(random_state=0), X_tr, y_tr, cv=5).mean()
print(f'training accuracy = {train_acc:.3f} vs cross-validated = {cv_acc:.3f} -> gap = {train_acc-cv_acc:.3f}')
training accuracy = 1.000 vs cross-validated = 0.670 -> gap = 0.330
Solution. An unlimited tree memorizes the training rows (accuracy near 1.0) but generalizes far worse, the gap is overfitting.
CHALLENGE 3
Cross-validate a model
Report the 5-fold CV mean and standard deviation for a depth-4 tree.
In [5]:
scores = cross_val_score(DecisionTreeClassifier(max_depth=4, random_state=0), X_tr, y_tr, cv=5)
print('folds:', np.round(scores,3), f'| mean {scores.mean():.3f} std {scores.std():.3f}')
folds: [0.743 0.721 0.686 0.721 0.764] | mean 0.727 std 0.026
Solution. Five folds give a stable mean and a spread; the standard deviation shows how much a single split would have wobbled.
CHALLENGE 4
Tune with GridSearchCV
Search max_depth and min_samples_leaf; report the best settings and CV score.
In [6]:
grid = GridSearchCV(DecisionTreeClassifier(random_state=0), {'max_depth':[2,3,4,5,6,8], 'min_samples_leaf':[1,10,20,40]}, cv=5).fit(X_tr, y_tr)
print('best params:', grid.best_params_, f'| best CV accuracy {grid.best_score_:.3f}')
best params: {'max_depth': 5, 'min_samples_leaf': 40} | best CV accuracy 0.740
Solution. The grid is searched with cross-validation, and the winning settings avoid both underfitting (too shallow) and overfitting (too deep).
CHALLENGE 5
Catch the leak
Add the post-outcome column and show cross-validated accuracy jumps unrealistically.
In [7]:
honest = cross_val_score(DecisionTreeClassifier(max_depth=4, random_state=0), X_tr, y_tr, cv=5).mean()
X_leak = X_tr.copy(); X_leak['sent_to_collections'] = df.loc[X_tr.index, 'sent_to_collections']
leaked = cross_val_score(DecisionTreeClassifier(max_depth=4, random_state=0), X_leak, y_tr, cv=5).mean()
print(f'honest {honest:.3f} vs with leak {leaked:.3f} <- too good to be true')
honest 0.727 vs with leak 0.916 <- too good to be true
Solution. sent_to_collections is recorded after default, so it nearly copies the label; it is unknowable at prediction time and must be excluded.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher