A gym owner knows bigger homes sell for more. Useful, but vague. Regression sharpens it to a sentence you can bank on: each extra square foot is worth about 180 dollars, and a 2,000 sqft home should sell for roughly 416,000 dollars, give or take. That is a line, a slope, and an interval, the whole of this chapter.
Fitting the line is the easy part. Doing regression well is a five-step discipline: check the data, fit the model, evaluate its adequacy, re-check the conditions on the residuals, then interpret and predict, looping back whenever a condition fails.
The Best-Practice Workflow
Anyone can call .fit(). The difference between a number you can trust and one that quietly misleads
is process. Every regression in this Part, simple, multiple, logistic, regularized, runs
through the same five steps. Learn them once here and you will apply them everywhere.
- 1Check the data. Before fitting, look. Is the relationship plausibly a straight line? Are the observations independent? Later you will confirm the residuals are roughly Normal with constant spread. These are the conditions summarized by LINE: Linearity, Independence, Normality, Equal variance.
- 2Define and fit the model. State which variable predicts which, then let least squares find the slope and intercept that minimize the squared residuals.
- 3Evaluate model adequacy. How much does the line explain (R²)? Is the slope significantly different from zero (t-test, confidence interval)? How large is the typical prediction error (residual standard error)?
- 4Check the conditions. Now interrogate the residuals with plots. A residual-versus-fitted plot with no pattern, and a roughly straight Q-Q plot, mean the LINE conditions hold. If not, you loop back.
- 5Interpret and predict. Only now do you read the slope in plain language and forecast, reporting a confidence interval for the average and a wider prediction interval for a single new case.
The dashed red arrow in the diagram is the part beginners skip: when a condition fails at step 4, you do not report the model anyway, you transform or re-specify and re-fit. We will do exactly that later in the chapter.
Fitting the Line and Reading It
Least squares slides a line through the cloud until the total squared vertical gap to the points is as small as
possible. In Python this is one line, ols("y ~ x", data).fit() from statsmodels, and it hands back
four numbers worth knowing by heart.
| Output | What it is | How to read it |
|---|---|---|
| Slope b₁ | Change in ŷ per one-unit rise in x | the effect size, and usually the whole point of the model |
| Intercept b₀ | Predicted y when x = 0 | meaningful only if x = 0 is in range; otherwise just anchors the line |
| R² | Share of the variation in y the line explains | 0 to 1; 0.80 means the line accounts for 80% of the spread in y |
| Residual std error | Typical size of a prediction miss | in the units of y; the smaller, the tighter the fit |
Two guardrails. First, the slope has uncertainty: statsmodels reports its standard error, a t-statistic, and a 95% confidence interval, so you can say whether the relationship is real or could be zero. Second, a high R² does not mean the model is correct. A curved relationship can still post a big R² while every prediction is biased. That is why step 4 exists, and why R² alone is never the finish line.
Checking Conditions, and What To Do When They Fail
The regression estimates are only as trustworthy as four conditions, the LINE assumptions. The good news: one plot checks most of them at once. Scatter the residuals against the fitted values and you want a shapeless, horizontal band around zero. A pattern is the model telling you something is wrong.
The left panel is what "conditions met" looks like: points scattered evenly in a flat band, no curve, no funnel. The right panel fails twice at once, a curve (the straight line missed a bend, so linearity is violated) and a funnel (the spread grows left to right, so equal variance is violated). When you see that, do not report the fit. Fix it:
| Condition | How you spot the violation | What to do about it |
|---|---|---|
| Linearity | Curve or U-shape in residuals vs fitted | add a squared term, transform x, or use a curved model (see Regularization & Flexible Models) |
| Independence | Residuals track over time or by group (Durbin-Watson far from 2) | add the grouping variable, or use time-series / panel methods (see Econometrics & Panel Data) |
| Normality | Q-Q plot bends at the ends; heavy tails | often mild; transform y, or lean on large-sample robustness |
| Equal variance | Funnel shape (heteroscedasticity); Breusch-Pagan p < 0.05 | log-transform y, use weighted least squares, or robust standard errors |
The most common single fix is a log transform of y. When a response grows
multiplicatively (prices, counts, incomes), its variance grows with its level, producing exactly that funnel.
Modeling log(y) straightens the curve and stabilizes the variance in one move. Here is that
remedy on a real, broken model, the same worked example the notebook builds:
On the left, the raw model's residuals fan out and fail the equal-variance test (Breusch-Pagan p < 0.001). On
the right, refitting on log(y) turns the funnel into a flat, healthy band that passes (p = 0.38),
without touching the data itself. This is step 4's feedback loop in action: diagnose the
failure, apply the matching remedy from the table above, and re-fit until the residuals look right.
Real-World Example: Predicting Home Prices
Let us run the full five-step workflow on real-estate data: predict a home's price from its
size_sqft. One predictor, one response, the purest simple regression.
One row per home sale with size_sqft, bedrooms,
age_years, neighborhood, and the sale price.
Steps 1–2, check and fit. A scatter of price against size is convincingly straight, so we fit
ols("price ~ size_sqft", homes).fit().
The fitted line threads cleanly through the cloud of homes, and the shaded band marks the small uncertainty about the average price at each size. Nothing curves and nothing fans out, the visual green light for a straight-line model before we trust a single number.
Step 3, evaluate. The model is strong and the numbers are clean:
| Result | Value | Reading |
|---|---|---|
| Slope (size_sqft) | $180 / sqft | each extra square foot adds about 180 dollars to the price |
| Slope 95% CI | $175 to $185 | the effect is precisely estimated and clearly not zero |
| Intercept | $56,000 | anchors the line; a literal "0 sqft" reading is meaningless extrapolation |
| R² | 0.965 | size alone explains about 96% of the variation in price |
| Residual std error | ≈ $27,500 | typical miss of a single prediction |
Step 4, check the conditions. The residual plots pass every test: the residual-versus-fitted band is flat (Breusch-Pagan p = 0.81, equal variance holds), the Q-Q plot is straight (Shapiro p = 0.86, residuals are Normal), and Durbin-Watson is 2.2 (independent). The LINE conditions are met, so we can trust the model and move on.
Step 5, interpret and predict. To predict, plug a size straight into the fitted line pricê = 56,041 + 180.13 × size_sqft. For a 2,000 sqft home:
That single number is the point prediction. But "give or take" splits into two very different intervals, and confusing them is a classic error:
| Interval at 2,000 sqft | Range | Answers the question |
|---|---|---|
| 95% confidence interval | $412k to $420k | where is the average price of all 2,000 sqft homes? |
| 95% prediction interval | $362k to $471k | what will this one 2,000 sqft home sell for? |
The confidence band (inner, orange) is narrow because averages are easy to pin down. The prediction band (outer, lighter) is far wider because a single home also carries the full residual scatter. Notice both bands are tightest at the center of the data and flare toward the extremes: the model is most confident near the average x and least confident at the edges. Report the confidence interval when you mean the trend, the prediction interval when you mean an individual, never mix them up.
Linear Regression in Machine Learning & AI
Simple linear regression is the atom of supervised learning. Every idea here scales directly into modern ML.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Least squares | The squared-error loss minimized by gradient descent | the training objective of linear and neural models |
| One predictor | Many features (see Multiple Linear Regression) and learned representations | the linear layer at the head of a deep network |
| R² / residual error | Validation metrics (RMSE, R²) on held-out data | train-test split, cross-validation |
| Prediction interval | Uncertainty quantification | conformal prediction, Bayesian regression |
| Checking conditions | Residual analysis & model diagnostics | catching bias a single accuracy score hides |
The five-step workflow is not a beginner's crutch you outgrow; it is what separates a reported number from a reliable one at every scale. A deep network trained with squared-error loss is doing least squares on learned features, and it can post a strong average score while its residuals reveal systematic bias on a subgroup, exactly the failure a residual plot exposes and an accuracy number hides. Uncertainty, too, carries over: the gap between a confidence and a prediction interval is the same distinction modern conformal prediction formalizes to put honest error bars on any model's output. Learn to check conditions on a two-variable line and you have the habit that keeps billion-parameter models honest.
Run the whole workflow in Python
The companion notebook fits the line with statsmodels, reads the slope, R², and confidence interval,
plots the fit with confidence and prediction bands, runs the residual diagnostics, and then
deliberately breaks a model (a fan-shaped, heteroscedastic example) to show the log-transform remedy. It
finishes on simple-linear-regression--homes.xlsx, walking all five steps end to end.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, statsmodels, and openpyxl and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Regression fits ŷ = b₀ + b₁x by least squares: the slope is the predicted change in y per one-unit rise in x.
- ✓Follow the five-step workflow: check the data, fit, evaluate adequacy, re-check conditions on the residuals, then interpret, looping back when a condition fails.
- ✓The LINE conditions (Linearity, Independence, Normality, Equal variance) are checked with residual plots; a curve or a funnel means fix the model, often with a log of y.
- ✓Confidence ≠ prediction interval: the confidence band covers the average response (narrow), the prediction band covers a single new case (wide); both flare away from the mean of x.
- ✓Real data: home price on size gives a slope near $180/sqft, R² ≈ 0.96, and a 2,000 sqft prediction of about $416,300.
Practice Challenges
Five short challenges, beginner to intermediate. Reach for statsmodels and scikit-learn, not hand-rolled formulas, before checking the solutions.
Fit and read a line
Fit price ~ size_sqft and report the slope, its 95% CI, and R² in one sentence each.
ols(...).fit(), then .params, .conf_int(), .rsquared.Confidence vs prediction interval
At 2,000 sqft, compute both intervals and explain in a sentence why one is much wider.
get_prediction(...).summary_frame() returns mean_ci and obs_ci columns.Diagnose the conditions
Plot residuals vs fitted and a Q-Q plot; run Breusch-Pagan and interpret whether LINE holds.
het_breuschpagan, sm.qqplot, durbin_watson.Break it, then fix it
Simulate y = exp(a + b·x + noise), fit the raw model, show the funnel, then refit on log(y).
Least squares by hand, then by library
Compute b₁ = r · (sy/sx) and b₀ = ẏ − b₁̄x, and confirm they match statsmodels.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on simple linear regression and the workflow. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can fit a line, judge it, check its conditions, and predict with honest error bars. The OLS Framework chapter opens the hood: the normal equations, the matrix form, and why least squares is the best linear unbiased estimator.