Ordinary least squares has one goal: fit the training data as closely as possible. Give it enough predictors or a high enough polynomial degree and it will fit them perfectly, and predict the future terribly, because it has memorized the noise. Regularization deliberately fits the data a little worse in exchange for generalizing a lot better.
When the workflow's diagnostics say overfitting (or too many, or collinear, predictors), regularize. When they say underfitting a curve, add flexibility with polynomials or splines. Either way, cross-validation picks the setting.
Overfitting and the Bias-Variance Trade-off
Every model lives on a spectrum between too simple and too complex. A too-simple model has high bias, it cannot represent the pattern (the straight line below misses the curve). A too-complex model has high variance, it changes wildly with the sample, chasing noise as if it were signal.
The only honest way to find the middle is to measure error on data the model did not train on. That is cross-validation: split the data into folds, train on most, test on the rest, rotate, and average. Training error always falls as complexity rises; validation error falls, then rises, and its lowest point is the model that will generalize best.
Ridge, Lasso, and Elastic Net
Instead of removing complexity by hand, let a penalty do it. Both methods add a term to the loss that grows with coefficient size, so the fit must justify every coefficient it keeps large:
That small difference, squared versus absolute, has a big consequence. Ridge shrinks all coefficients toward zero but rarely to zero. Lasso's corner geometry lets it set coefficients exactly to zero, so it performs automatic feature selection. Watch the lasso do it as the penalty grows:
Reading right to left as the penalty strengthens, the three noise features are pinned to zero almost immediately,
while the genuine predictors (temperature², soil quality, rainfall, fertilizer) survive far longer. The
dashed purple line is the λ that cross-validation chose. When predictors are correlated, ridge
is often steadier (it keeps and shrinks the whole group), while elastic net blends
the two penalties to get selection and stability. All three are one line in scikit-learn:
RidgeCV, LassoCV, ElasticNetCV.
Flexible Models: Polynomials, Splines, and Choosing Complexity
Regularization fights overfitting; the opposite problem, a straight line through a curved relationship, is fixed by adding flexibility. Polynomial regression adds x², x³, … terms; splines stitch together local polynomials for flexibility that does not explode at the edges the way a high-degree polynomial does. Both are still linear regression, just on transformed features, so the whole workflow still applies.
The catch is the same trade-off: too few terms underfit, too many overfit. Cross-validation draws the map:
Training error slides toward zero as you add terms, but validation error traces a U, and its bottom is the sweet spot. In the crop example, a degree-2 polynomial in temperature lifts cross-validated R² from 0.17 (a straight line) to 0.65, while degrees beyond 2 add nothing but wiggle. The rule of thumb: the simplest model within reach of the best validation score wins.
Real-World Example: Predicting Crop Yield
We predict crop yield from weather, inputs, soil, and three deliberately irrelevant measurements, a
perfect test of whether a method can find the signal and ignore the noise.
One row per field plot with temperature, rainfall,
fertilizer, soil_quality, three noise columns, and yield.
Step 1, check the data. Plot the response against the key predictor. Yield does not rise in a straight line, it peaks at a moderate temperature and falls off on either side, an inverted U:
A straight line would slice through the middle and miss both the cold and hot plots, so it badly underfits (a cross-validated R² of only 0.17). The degree-2 curve tracks the bend and lifts CV R² to 0.65, that gap is the price of too little flexibility.
Step 2, fit. With the curve captured (a squared temperature term) and every feature standardized, we fit three models, ordinary least squares, ridge, and lasso, on all eight predictors, including three deliberate noise columns.
Step 3, evaluate. All three reach R² ≈ 0.91, but only the lasso cleans house:
| Feature | OLS coef | Lasso coef | Outcome |
|---|---|---|---|
| temperature² | −8.60 | −8.37 | kept, the dominant driver |
| soil_quality | 2.94 | 2.85 | kept |
| rainfall | 2.31 | 2.18 | kept |
| fertilizer | 1.68 | 1.60 | kept |
| noise1, noise3 | −0.05, −0.05 | 0.00 | zeroed out |
OLS gives the noise features small but nonzero coefficients, pretending they matter. Lasso sets them to exactly zero, handing you a shorter, more honest model with no loss of accuracy.
Step 4, check the conditions. The condition to watch here is generalization, not a residual plot, too little flexibility underfits, too much overfits. Cross-validation is the diagnostic (train on part, score on the held-out rest), and the penalty, ridge, lasso, or elastic net, with λ chosen by CV, is the remedy: a built-in defense against overfitting when the real data are noisier than this.
Step 5, interpret and predict. The degree-2 temperature model is a quadratic you can evaluate by hand:
At the optimal 24°C: −0.088×576 + 4.22×24 − 12.8 ≈ 37.9 tons/ha, the top of the curve.
Regularization in Machine Learning & AI
Regularization is not a regression footnote, it is one of the load-bearing ideas of modern machine learning.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Ridge (L2) penalty | Weight decay | the default regularizer in nearly every neural network |
| Lasso (L1) penalty | Sparsity / feature selection | compact models, sparse attention, pruning |
| Bias-variance trade-off | The central tuning problem | capacity vs data; double descent |
| Cross-validation | Model selection & hyperparameter tuning | choosing depth, width, learning rate |
| Splines / basis expansion | Feature learning & kernels | a network learns its own flexible basis |
The single most common line in a deep-learning training loop after the loss is weight decay, and weight decay is ridge regression's L2 penalty, applied to millions of parameters. Dropout, early stopping, and data augmentation are all regularizers too, different mechanisms for the same goal this chapter names: trade a little training fit for a lot of generalization. The bias-variance trade-off is the lens through which model capacity is chosen everywhere, and its modern twist, double descent, where error falls again past the interpolation point, is a live research frontier that only makes sense once you understand the classic U-curve here. Master ridge and lasso on eight features and you have the concepts that govern models with eight billion.
Tune the complexity dial in Python
The companion notebook plots the bias-variance curve across polynomial degrees, fits ridge,
lasso, and elastic net with scikit-learn, draws the regularization path, uses
cross-validation to pick λ, and shows the lasso zeroing the noise features on
regularization-and-flexible-models--crops.xlsx, all with a few library calls.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, scikit-learn, and openpyxl and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Two failure modes: high bias (underfit) and high variance (overfit); the goal is the model at the bottom of the validation U-curve.
- ✓Ridge (L2) shrinks coefficients smoothly and stabilizes correlated predictors; lasso (L1) zeros some coefficients, doing automatic feature selection.
- ✓Elastic net blends both; the penalty λ is set by cross-validation (
RidgeCV,LassoCV,ElasticNetCV). - ✓Add flexibility for underfitting with polynomials or splines, still linear regression on transformed features, and choose the degree by cross-validation.
- ✓Real data: a degree-2 term lifted CV R² from 0.17 to 0.65, and lasso zeroed the three noise features while OLS, ridge, and lasso all reached R² ≈ 0.91.
Practice Challenges
Five short challenges with scikit-learn pipelines and cross-validation.
Draw the bias-variance curve
Fit polynomials of degree 1 to 15 for temperature → yield; plot train vs cross-validated R².
make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).Ridge vs OLS on correlated features
Compare OLS and ridge coefficients for the correlated fertilizer / soil pair.
RidgeCV.Lasso feature selection
Fit LassoCV on all features and list which ones survive with nonzero coefficients.
Cross-validate the penalty
Report the λ each of RidgeCV and LassoCV selects, and the resulting cross-validated R².
.alpha_ and cross_val_score.Polynomial vs spline
Fit a degree-2 polynomial and a spline to temperature → yield; compare cross-validated fit.
SplineTransformer in a pipeline.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on regularization and flexible models. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now bend a model to fit a curve and rein it in to avoid overfitting. Generalized Linear Models unifies everything in this Part, linear, logistic, and more, under one elegant umbrella of link functions.