Chapter 108 · Solutions
Ensemble Methods · Solutions
Five challenges, each verified in code.
Chapter 108 · Practice Challenges, Solved¶
Five exercises on ensembles, worked in scikit-learn on the Chapter 108 attrition table.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, GradientBoostingClassifier
try: df = pd.read_excel('../../data/ensemble-methods--attrition.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'ensemble-methods--attrition.xlsx', sheet_name='Data')
feat = ['satisfaction','last_evaluation','num_projects','avg_monthly_hours','tenure_years','work_accident','promotion_5yr','salary_level','noise_a','noise_b']
X, y = df[feat], df['left']
print('loaded', df.shape)
loaded (1000, 12)
CHALLENGE 1
A single tree overfits
Show a full tree scores ~1.0 on training but far less in cross-validation.
In [3]:
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
t = DecisionTreeClassifier(random_state=0).fit(X_tr, y_tr)
print(f'train {t.score(X_tr,y_tr):.3f} vs CV {cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean():.3f}')
train 1.000 vs CV 0.704
Solution. The perfect training score with a much lower CV score is memorization, the overfitting an ensemble cures.
CHALLENGE 2
Random forest and importances
Fit a random forest, report its CV accuracy, and name the top feature.
In [4]:
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X, y)
print(f'CV accuracy = {cross_val_score(RandomForestClassifier(n_estimators=300, random_state=0), X, y, cv=5).mean():.3f}')
imp = pd.Series(rf.feature_importances_, index=feat).sort_values(ascending=False)
print('top feature:', imp.idxmax(), f'({imp.max():.3f})'); print('noise importances:', imp[['noise_a','noise_b']].round(3).to_dict())
CV accuracy = 0.782
top feature: satisfaction (0.376)
noise importances: {'noise_a': 0.104, 'noise_b': 0.099}
Solution. The forest lifts accuracy well above the single tree and correctly ranks the noise columns near zero.
CHALLENGE 3
Bagging reduces variance
Compare a single tree to a 300-tree bagging ensemble by CV accuracy.
In [5]:
print(f'single tree : {cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean():.3f}')
print(f'bagging(300): {cross_val_score(BaggingClassifier(n_estimators=300, random_state=0), X, y, cv=5).mean():.3f}')
single tree : 0.704
bagging(300): 0.777
Solution. Averaging many bootstrap trees cancels the variance of any single one, so accuracy climbs.
CHALLENGE 4
Boosting vs the forest
Compare gradient boosting to the random forest by CV accuracy.
In [6]:
print(f'random forest : {cross_val_score(RandomForestClassifier(n_estimators=300, random_state=0), X, y, cv=5).mean():.3f}')
print(f'gradient boosting: {cross_val_score(GradientBoostingClassifier(random_state=0), X, y, cv=5).mean():.3f}')
random forest : 0.782
gradient boosting: 0.789
Solution. Both are strong; boosting (sequential, bias-reducing) often edges out the forest on structured tabular data.
CHALLENGE 5
The ladder
Rank single tree, bagging, random forest, and gradient boosting by CV accuracy.
In [7]:
models = {'tree': DecisionTreeClassifier(random_state=0), 'bagging': BaggingClassifier(n_estimators=300, random_state=0),
'forest': RandomForestClassifier(n_estimators=300, random_state=0), 'boosting': GradientBoostingClassifier(random_state=0)}
print(pd.Series({k: cross_val_score(m, X, y, cv=5).mean() for k,m in models.items()}).sort_values().round(3).to_string())
tree 0.704 bagging 0.777 forest 0.782 boosting 0.789
Solution. The lone tree is worst; every ensemble beats it, and boosting tops the ladder here.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher