Chapter 116 · Solutions
Model Pitfalls · Solutions
Five challenges, each verified in code.
Chapter 116 · Practice Challenges, Solved¶
Five exercises on model pitfalls, worked in scikit-learn and imbalanced-learn.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, validation_curve, KFold, StratifiedKFold, cross_val_score
from sklearn.metrics import mean_squared_error, recall_score
rng = np.random.default_rng(1)
xt = np.sort(rng.uniform(0,1,40)); yt = np.sin(2*np.pi*xt) + rng.normal(0,0.25,40)
print('setup ready')
setup ready
CHALLENGE 1
Underfit vs overfit
Fit a degree-1 and a degree-15 polynomial; compare training and test error.
In [3]:
xa,xb,ya,yb = train_test_split(xt, yt, test_size=0.4, random_state=0)
for d in [1,15]:
m=make_pipeline(PolynomialFeatures(d),LinearRegression()).fit(xa.reshape(-1,1),ya)
print(f'degree {d:2d}: train MSE {mean_squared_error(ya,m.predict(xa.reshape(-1,1))):.3f} test MSE {mean_squared_error(yb,m.predict(xb.reshape(-1,1))):.3f}')
degree 1: train MSE 0.220 test MSE 0.275 degree 15: train MSE 0.013 test MSE 1.548
Solution. Degree 1 underfits, with both errors high. Degree 15 overfits, with a tiny training error and a test error more than 100 times larger, well above degree 1's. With only 16 test points one split is a noisy judge, which is why the next challenge cross-validates.
CHALLENGE 2
Find the sweet spot
Use a validation curve to pick the polynomial degree with the lowest validation error.
In [4]:
tr,va = validation_curve(make_pipeline(PolynomialFeatures(1),LinearRegression()), xt.reshape(-1,1), yt, param_name='polynomialfeatures__degree', param_range=list(range(1,16)), cv=KFold(5, shuffle=True, random_state=0), scoring='neg_mean_squared_error')
best = list(range(1,16))[int(np.argmin(-va.mean(axis=1)))]
print('best degree (lowest validation error) =', best)
best degree (lowest validation error) = 5
Solution. Validation error is U-shaped; its minimum is the bias-variance sweet spot.
CHALLENGE 3
Diagnose with a gap
Show that a high-degree model has a large train-validation error gap (overfitting).
In [5]:
m=make_pipeline(PolynomialFeatures(14),LinearRegression())
tr=cross_val_score(m,xt.reshape(-1,1),yt,cv=KFold(5, shuffle=True, random_state=0),scoring='neg_mean_squared_error')
m.fit(xt.reshape(-1,1),yt); train_mse=mean_squared_error(yt,m.predict(xt.reshape(-1,1)))
print(f'training MSE {train_mse:.3f} vs CV MSE {-tr.mean():.3f} -> big gap = overfitting')
training MSE 0.024 vs CV MSE 0.147 -> big gap = overfitting
Solution. A low training error beside a much higher validation error is the fingerprint of high variance.
CHALLENGE 4
Rebalance a rare class
On the defect data, compare recall with and without class_weight='balanced'.
In [6]:
try: df=pd.read_excel('../../data/model-pitfalls--defects.xlsx',sheet_name='Data')
except FileNotFoundError: df=pd.read_excel(BASE+'model-pitfalls--defects.xlsx',sheet_name='Data')
feat=['temperature','pressure','line_speed','machine_age_mo','material_grade']; X,y=df[feat],df['defective']
Xtr,Xte,ytr,yte=train_test_split(X,y,test_size=0.3,random_state=0,stratify=y)
for cw in [None,'balanced']:
m=make_pipeline(StandardScaler(),LogisticRegression(max_iter=1000,class_weight=cw)).fit(Xtr,ytr)
print(f'class_weight={str(cw):9s} recall {recall_score(yte,m.predict(Xte)):.3f}')
class_weight=None recall 0.074 class_weight=balanced recall 0.815
Solution. Balancing the classes lifts recall on the rare defect class dramatically (at some cost in precision).
CHALLENGE 5
Stratify the folds
Show StratifiedKFold keeps each fold's defect rate close to the overall rate, while plain KFold does not.
In [7]:
print('KFold :', [round(float(y.iloc[te].mean()),3) for _,te in KFold(5,shuffle=True,random_state=1).split(X)])
print('Stratified :', [round(float(y.iloc[te].mean()),3) for _,te in StratifiedKFold(5,shuffle=True,random_state=1).split(X,y)])
KFold : [0.083, 0.075, 0.096, 0.062, 0.054] Stratified : [0.071, 0.075, 0.075, 0.075, 0.075]
Solution. Stratification preserves the class ratio in every fold, giving a stable, representative CV estimate on imbalanced data.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher