←Back to the bookChapter 108 · Ensemble Methods

Chapter 108 · Ensemble Methods

Ensemble Methods

Combining many weak learners into a strong one on an employee-attrition table: bagging and the random forest for variance, boosting (AdaBoost, gradient boosting, and the XGBoost / LightGBM / CatBoost family) for bias, and stacking, all library-first, visual, and cross-validated.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

Ensemble Methods: many weak learners, one strong model

A single decision tree is easy to read but easy to fool: too shallow and it underfits, too deep and it memorizes. Ensembles fix this by combining many trees. This notebook builds bagging and a random forest (average many trees to cut variance), boosting (grow trees in sequence, each fixing the last one's mistakes, to cut bias), and stacking, then races them on an employee-attrition table. Library-first with scikit-learn.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (BaggingClassifier, RandomForestClassifier, GradientBoostingClassifier,
    AdaBoostClassifier, StackingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import accuracy_score
pd.set_option('display.max_columns', 30)

DEMO 1 · THE BASE LEARNER OVERFITS

One tree is not enough

A single unpruned tree memorizes the training data.

try: df = pd.read_excel('../../data/ensemble-methods--attrition.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'ensemble-methods--attrition.xlsx', sheet_name='Data')
feat = ['satisfaction','last_evaluation','num_projects','avg_monthly_hours','tenure_years',
        'work_accident','promotion_5yr','salary_level','noise_a','noise_b']
X, y = df[feat], df['left']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(f'single full tree: train {tree.score(X_train,y_train):.3f}  vs  5-fold CV {cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean():.3f}')
print(f'attrition rate {y.mean():.0%}  (majority-class baseline {1-y.mean():.3f})')

single full tree: train 1.000  vs  5-fold CV 0.704
attrition rate 40%  (majority-class baseline 0.602)

The problem. A single unrestricted decision tree scores a perfect 1.00 on its training data but only about 0.70 in cross-validation, worse than a plain logistic regression. It has memorized noise (including the two junk noise columns) instead of learning the pattern. The attrition risk here is genuinely nonlinear, a three-way interaction of high performers who are overworked and never promoted, so one tree cannot win. Ensembles are the fix.

DEMO 2 · BAGGING & THE RANDOM FOREST

Average many trees to cut variance

Each tree sees a bootstrap sample; the forest votes.

bag = BaggingClassifier(n_estimators=300, random_state=0)
rf  = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0).fit(X_train, y_train)
print(f'bagging      5-fold CV = {cross_val_score(bag, X, y, cv=5).mean():.3f}')
print(f'random forest 5-fold CV = {cross_val_score(RandomForestClassifier(n_estimators=300, random_state=0), X, y, cv=5).mean():.3f}')
print(f'random forest OOB score = {rf.oob_score_:.3f}   (free validation from the un-sampled rows)')
imp = pd.Series(rf.feature_importances_, index=feat).sort_values()
fig, ax = plt.subplots(figsize=(7.2,4.4))
ax.barh(imp.index, imp.values, color=[GREY if 'noise' in f else EM for f in imp.index])
ax.set(title='Random-forest feature importances (gray = the junk noise columns)', xlabel='importance'); plt.tight_layout(); plt.show()

bagging      5-fold CV = 0.777

random forest 5-fold CV = 0.782
random forest OOB score = 0.793   (free validation from the un-sampled rows)

What the code does. Bagging (bootstrap aggregating) trains each tree on a random resample of the rows and averages their votes, which cancels out the variance that made one tree unstable. A random forest adds a second dose of randomness, each split considers only a random subset of features, so the trees disagree more and average better. Cross-validated accuracy jumps to about 0.78. The OOB score is a free validation estimate from the rows each tree did not see. And the feature importances correctly rank satisfaction, hours, and evaluation at the top while pushing the two noise columns to nearly zero, something the single tree failed to do.

DEMO 3 · SMOOTHER DECISION BOUNDARY

The forest generalizes where the tree overfits

Trained on two features so we can see it.

X2 = df[['satisfaction','avg_monthly_hours']].values; yv = df['left'].values
xx, yy = np.meshgrid(np.linspace(X2[:,0].min(), X2[:,0].max(), 250), np.linspace(X2[:,1].min(), X2[:,1].max(), 250))
fig, axes = plt.subplots(1, 2, figsize=(11,4.4))
for ax,(name,m) in zip(axes, [('single full tree (overfit)', DecisionTreeClassifier(random_state=0)), ('random forest (smooth)', RandomForestClassifier(n_estimators=300, random_state=0))]):
    m.fit(X2, yv)
    Z = m.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.25, cmap=ListedColormap([LIGHT, '#fca5a5']))
    ax.scatter(X2[:,0], X2[:,1], c=yv, cmap=ListedColormap([EM, RED]), s=8, alpha=0.4)
    ax.set(title=name, xlabel='satisfaction', ylabel='avg monthly hours')
plt.suptitle('Blue = stay, red = leave', y=1.02); plt.tight_layout(); plt.show()

Reading the plot. The single tree carves the plane into tiny jagged rectangles, chasing individual points (overfitting). The random forest, by averaging hundreds of such trees, produces a smooth, sensible boundary that captures the real regions (low satisfaction, and very high hours) without the noise. Same data, far better generalization, the visual signature of variance reduction.

DEMO 4 · BOOSTING

Grow trees in sequence, each fixing the last

Boosting reduces bias; watch accuracy climb as trees are added.

gb = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)
ada = AdaBoostClassifier(n_estimators=200, random_state=0)
print(f'gradient boosting 5-fold CV = {cross_val_score(GradientBoostingClassifier(random_state=0), X, y, cv=5).mean():.3f}')
print(f'AdaBoost          5-fold CV = {cross_val_score(ada, X, y, cv=5).mean():.3f}')
acc = [accuracy_score(y_test, p) for p in gb.staged_predict(X_test)]
fig, ax = plt.subplots(figsize=(7.4,4))
ax.plot(range(1, len(acc)+1), acc, color=EM, lw=2)
ax.set(title='Gradient boosting: test accuracy climbs as trees are added', xlabel='number of boosting stages (trees)', ylabel='test accuracy')
plt.tight_layout(); plt.show()

gradient boosting 5-fold CV = 0.789

AdaBoost          5-fold CV = 0.791

How boosting differs. Bagging builds its trees in parallel and independently; boosting builds them in sequence, where each new (shallow) tree focuses on the examples the previous ones got wrong. This attacks bias rather than variance, and on this data gradient boosting is the top scorer (~0.79). The staged curve shows accuracy rising then leveling as stages accumulate; too many stages eventually overfit, so the number of trees and the learning rate are the key hyperparameters. The production-grade boosting libraries, XGBoost, LightGBM, and CatBoost, are highly optimized versions of exactly this idea and dominate tabular-data competitions.

DEMO 5 · STACKING

Let a meta-model combine the others

Blend diverse models with a final learner.

stack = StackingClassifier(
    estimators=[('rf', RandomForestClassifier(n_estimators=200, random_state=0)),
                ('gb', GradientBoostingClassifier(random_state=0)),
                ('lr', make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)))],
    final_estimator=LogisticRegression(), cv=5)
print(f'stacking 5-fold CV = {cross_val_score(stack, X, y, cv=5).mean():.3f}')

stacking 5-fold CV = 0.789

What stacking does. Instead of a simple vote, stacking trains a small meta-model to learn how much to trust each base model, using their cross-validated predictions as inputs. It shines when the base learners are diverse (here a forest, a booster, and a linear model make different kinds of error), so the meta-model can combine their strengths. It is the technique behind many winning competition entries, at the cost of more compute and less interpretability.

DEMO 6 · THE FULL RACE

From one tree to the ensembles

Cross-validated accuracy, worst to best.

models = {'single tree (full)': DecisionTreeClassifier(random_state=0),
          'logistic regression': make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
          'bagging': BaggingClassifier(n_estimators=300, random_state=0),
          'random forest': RandomForestClassifier(n_estimators=300, random_state=0),
          'AdaBoost': AdaBoostClassifier(n_estimators=200, random_state=0),
          'gradient boosting': GradientBoostingClassifier(random_state=0),
          'stacking': stack}
cv = pd.Series({k: cross_val_score(m, X, y, cv=5).mean() for k,m in models.items()}).sort_values()
print(cv.round(3).to_string())
fig, ax = plt.subplots(figsize=(7.6,4.2))
colors = ['#cbd5e1' if 'tree' in k or 'logistic' in k else EM for k in cv.index]
ax.barh(cv.index, cv.values, color=colors)
ax.axvline(1-y.mean(), color=GREY, ls='--', label=f'baseline {1-y.mean():.2f}')
for i,v in enumerate(cv.values): ax.text(v+0.004, i, f'{v:.3f}', va='center', fontsize=9)
ax.set(title='Ensembles beat the single tree', xlabel='5-fold CV accuracy', xlim=(0.55, 0.85)); ax.legend(loc='lower right')
plt.tight_layout(); plt.show()

single tree (full)     0.704
logistic regression    0.770
bagging                0.777
random forest          0.782
gradient boosting      0.789
stacking               0.789
AdaBoost               0.791

The verdict. The lone overfit tree sits at the bottom; every ensemble lands at the top, with boosting and stacking edging out the forest. The lesson is not that one ensemble always wins, but that combining many models beats relying on one, either by averaging away variance (bagging / random forest) or by sequentially reducing bias (boosting). For tabular data, a tuned gradient-boosting model is very often the strongest thing you can reach for.

Ensembles, in one view

• Bagging trains many models on bootstrap resamples and averages them, cutting variance; the random forest adds random feature subsets so the trees are more independent.
• Boosting grows models sequentially, each correcting the last, cutting bias; AdaBoost and gradient boosting (XGBoost / LightGBM / CatBoost) are the workhorses.
• Stacking trains a meta-model to blend diverse base models.
• Ensembles trade interpretability and compute for accuracy; on tabular data they are usually the top performers, which is why boosted trees dominate competitions.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 108 · Ensemble Methods