One expert can be wrong; a diverse committee usually is not. That is the whole idea of an ensemble: train many models and combine them, and the crowd's answer is more accurate and more stable than any single member. Applied to decision trees, this simple trick produces the strongest off-the-shelf models for tabular data.
A single tree overfits; combine many. Bagging / random forests average independent trees to reduce variance, boosting chains trees to reduce bias, and both beat the lone tree, with gradient boosting usually on top.
Why Combine Models?
The Core Classification & Regression Algorithms chapter ended with a single decision tree trailing every other algorithm. The reason is a fundamental tension: a shallow tree underfits (too simple to catch the pattern) while a deep tree overfits (it memorizes noise). In the notebook, an unrestricted tree scores a perfect 1.00 on its training data but only 0.70 in cross-validation, worse than plain logistic regression.
Ensembles escape the trap by combining many trees so their individual errors cancel. There are two fundamentally different ways to do it, and the distinction, parallel versus sequential, is the key to the whole chapter.
Bagging & Random Forests
Bagging (bootstrap aggregating) trains each tree on a different random resample of the rows, then averages their votes. Because the trees are trained independently, their random errors partly cancel, so the average is far more stable than any single tree. The random forest adds a second source of randomness: at every split, each tree may only consider a random subset of the features. That forces the trees to be more different from one another, which makes the averaging even more effective.
In the notebook, a random forest lifts cross-validated accuracy from the single tree's 0.70 to about 0.78,
and its decision boundary is smooth where the single tree's was a jagged patchwork of overfit rectangles. Two bonuses
come for free: the out-of-bag (OOB) score is a validation estimate from the rows each tree did not
sample, and the feature importances rank which inputs matter, here satisfaction, monthly hours, and
evaluation lead, while the two planted noise columns are correctly pushed to near zero.
Boosting
Boosting takes the opposite approach. Instead of many independent trees, it grows them one at a time, each new (shallow) tree focusing on the examples the previous ones got wrong. Where bagging attacks variance, boosting attacks bias, steadily turning a sequence of weak learners into a strong one. The notebook's staged-accuracy curve shows test accuracy climbing as trees are added, then leveling off (add too many and it eventually overfits, so the number of trees and the learning rate are the dials to tune).
The classic algorithm is AdaBoost; the modern default is gradient boosting, which fits each new tree to the residual errors of the running total. On this data, gradient boosting is the single best model (~0.79). And the three names you will meet everywhere in practice, XGBoost, LightGBM, and CatBoost, are all highly optimized gradient-boosting libraries; they are the tools that win most tabular-data competitions.
Real-World Example: Predicting Employee Attrition
An HR team has 1,000 employees and wants to predict who will leave. The risk is deliberately nonlinear, high performers who are overworked and never promoted, which is exactly the kind of interaction a single tree fumbles and an ensemble nails.
One row per employee with satisfaction, last_evaluation,
num_projects, avg_monthly_hours, tenure_years, salary_level,
two junk noise columns, and the label left (1 = left, 0 = stayed).
Racing every model with cross-validation puts the story in one picture:
The lone overfit tree sits at the bottom; every ensemble clusters at the top, with boosting and stacking (a meta-model that learns how much to trust each base model) edging out the forest. The takeaway is not that one ensemble always wins, but that combining many models beats relying on one, and for messy, interaction-heavy tabular data a tuned gradient-boosting model is usually the best thing you can grab.
Ensembles in Machine Learning & AI
Ensembles are not a niche trick, they are the default winning strategy across applied machine learning.
| Method (this chapter) | Where it dominates | Example |
|---|---|---|
| Random forest | Strong low-tuning baseline | the first model to try on tabular data; feature importances |
| Gradient boosting | State of the art on tables | XGBoost / LightGBM / CatBoost, most Kaggle tabular wins |
| Bagging / model averaging | Variance reduction | averaging several neural nets; snapshot ensembles |
| Boosting | Bias reduction, ranking | learning-to-rank in search, fraud and risk scoring |
| Stacking | Squeezing out the last % | competition-winning blended solutions |
Two big lessons live here. First, on tabular data, ensembles of decision trees, especially gradient boosting, still routinely beat deep neural networks; reaching for a giant model when a boosted-tree ensemble would win is a common and expensive mistake. Second, the ensembling idea is everywhere in deep learning too: averaging several trained networks, snapshot ensembles, and even dropout (which randomly disables neurons during training) all act as implicit ensembles that reduce variance. Whenever a system needs to be more accurate and more robust, combining diverse models is one of the most reliable levers you have.
Build every ensemble in Python
The companion notebook shows the single tree overfitting, then builds bagging and a random forest (with OOB score and feature importances), contrasts their decision boundaries, grows a gradient-boosting model with a staged-accuracy curve, blends a stacking ensemble, and races them all, with every step narrated.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓A single tree overfits (perfect on training, poor on new data); ensembles combine many trees to fix it.
- ✓Bagging averages independent bootstrap-trained trees to cut variance; the random forest adds random feature subsets.
- ✓Boosting grows trees sequentially, each fixing prior errors, to cut bias; gradient boosting is the modern default.
- ✓XGBoost, LightGBM, CatBoost are optimized gradient boosters, the champions of tabular ML; stacking blends diverse models with a meta-learner.
- ✓Ensembles win: on tabular data boosted trees often beat deep nets, and ensembling (model averaging, dropout) pervades deep learning too.
Practice Challenges
Five short challenges on the attrition table. Try them with scikit-learn before checking the solutions.
A single tree overfits
Show a full tree scores near 1.0 on training but far less in cross-validation.
.score on train vs cross_val_score.Random forest & importances
Fit a random forest, report its CV accuracy, and name the top feature.
rf.feature_importances_; the noise columns should be near zero.Bagging reduces variance
Compare a single tree to a 300-tree bagging ensemble by CV accuracy.
BaggingClassifier(n_estimators=300).Boosting vs the forest
Compare gradient boosting to the random forest by CV accuracy.
GradientBoostingClassifier().The ladder
Rank single tree, bagging, random forest, and gradient boosting by CV accuracy.
cross_val_score(m, X, y, cv=5).mean() and sort.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on ensemble methods. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
The random forest ranked its features and flagged the junk ones. Feature Selection makes that a discipline, filter, wrapper, and embedded methods for keeping the inputs that matter and dropping the ones that only add noise.