Multiple Linear Regression

A car's fuel economy depends on its weight, its power, and where it was built, all at once. Fit them one at a time and each steals credit from the others. Fit them together and multiple regression separates the signal: how much does weight matter once power and origin are already accounted for?

This four-word phrase is the heart of multiple regression. A partial slope answers a ceteris paribus question: compare two cars of the same power and origin but differing by one pound, the coefficient on weight is the expected mpg difference between them. That is why a predictor's coefficient can shrink, or even flip sign, when you add a related variable, you are now asking a different question.

To judge which predictors truly matter, plot each coefficient with its confidence interval. A whisker that clears zero is a real effect; one that straddles zero could be nothing.

Weight dominates. The origin effects are real and modest. But horsepower and displacement have intervals that cross zero, not because they do not affect mpg, but because they are so entangled with weight that the model cannot give them separate credit. That is your first sight of multicollinearity, diagnosed in Section 3. One more guardrail: use adjusted R², not plain R², to compare models, because plain R² can only rise when you add predictors, even useless ones.

Regression needs numbers, so a category like origin becomes a set of dummy variables: 0/1 columns, one per level, with one level held out as the reference. Each dummy's coefficient is that group's shift relative to the reference. Geometrically, dummies give every group the same slope but a different intercept, parallel lines.

Here Europe is the reference; Japan sits about 1.4 mpg above it and the USA about 1.7 below, at any weight. When the effect of one predictor depends on another, you need an interaction term (weight * C(origin)), which lets each group have its own slope, non-parallel lines. Test it before you trust it: in this dataset the interaction is not significant (p ≈ 0.20), so parallel lines are the honest description, adding the interaction would just fit noise.

When predictors are strongly correlated with each other, the model struggles to divide credit between them. The coefficients become unstable, their standard errors balloon, and individually significant variables can look worthless, even while the model as a whole predicts beautifully. The standard gauge is the variance inflation factor (VIF): how much a coefficient's variance is inflated by its correlation with the other predictors.

A rough rule: VIF above 5 deserves a look, above 10 is serious. In the car data, weight, horsepower, and displacement are all measuring "how big is the engine and body," so displacement's VIF is 13. The remedies are simple: drop the most redundant predictor, combine them into one index, or use ridge regression (see the Regularization & Flexible Models chapter), which is built for exactly this. Crucially, multicollinearity does not bias predictions, if you only care about forecasting mpg, you can ignore it. It only muddies the interpretation of individual coefficients.

We model mpg from weight, horsepower, displacement, and origin, then walk the workflow: fit, check for multicollinearity, fix it, and interpret.

We follow the same five-step workflow as the last two chapters, now with the multicollinearity check folded into steps 1 and 4.

Step 1, check the data. Before fitting anything, look at how mpg relates to each numeric predictor:

All three panels slope down, heavier, more powerful, larger-engined cars burn more fuel, so every predictor looks useful. But notice they tell almost the same story: weight, horsepower, and displacement are strongly correlated with one another (r = 0.87 to 0.94), because all three really measure "how big is the car." That overlap is the seed of the multicollinearity we will diagnose in step 4.

Step 2, define and fit. We regress mpg on all four predictors at once, ols("mpg ~ weight + horsepower + displacement + C(origin)", cars).fit(), so every coefficient is read with the others held constant.

Step 3, evaluate adequacy. The model explains the data well, adjusted R² = 0.90, but the coefficient table has a tell:

Weight and both origin effects are clearly significant, yet horsepower (p = 0.19) and displacement (p = 0.78) look worthless, even though step 1 showed each is strongly related to mpg. That contradiction is the fingerprint of multicollinearity: the predictors are so entangled the model cannot give any one of them separate credit. The coefficient forest plot in Section 1 shows exactly this, weight's interval sits far from zero while horsepower's and displacement's intervals straddle it.

Step 4, check the conditions. The variance inflation factors confirm the diagnosis, displacement 13.3, weight 8.3, horsepower 6.6 (above 10 is serious). The remedy is to drop the most redundant predictors and re-fit:

Dropping displacement and horsepower barely moves the fit (adjusted R² 0.902 to 0.901) but makes the weight coefficient stable and its interval tight, exactly what VIF told us to expect.

Step 5, interpret and predict. Read the surviving coefficients "holding the others constant", each pound of weight costs about 0.008 mpg, and Japanese cars average 1.4 mpg more (US cars 1.7 mpg less) than European. Then predict by plugging into the fitted line:

A 3,000 lb Japanese car (origin +1.41): 60.15 − 0.00805×3,000 + 1.41 ≈ 37.4 mpg. Across the whole fleet, predicted mpg tracks actual mpg tightly, the dots hug the dashed "perfect prediction" line:

The lesson in one line: dropping two redundant predictors cost essentially nothing in fit (0.902 to 0.901) and bought a model you can actually explain. Simpler, when it predicts just as well, wins.

Multiple regression is the direct ancestor of supervised learning with many features.

View opens the rendered notebook instantly (no setup). Open in Colab runs & edits it live in your browser. To run locally, install numpy, pandas, scipy, matplotlib, seaborn, statsmodels, and openpyxl and launch jupyter notebook.

Five short challenges. Use statsmodels formulas (C(...), *) and the VIF helper.

Eight quick questions on multiple regression. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.