Part XV taught how to fit a regression and read what came out. This part is about the four decisions that fix what the output means before anybody reads it, each of which can be made wrongly by a model that fits beautifully and reports nothing amiss.
Four projects, each built around one of the four decisions. In every one of them the model fits, the software reports no error, and the naive answer is wrong: by a factor of six, by the deletion of the finding, by a factor of two in predicted risk, and by a reversed sign.
What This Part Adds
Part XV covered the mechanics: fit the model, check the residuals, read the coefficient with its interval. Those five steps still apply and are not repeated here. What Part XV largely assumed is the part that goes wrong most often in practice.
| Part XV asked | This part asks |
|---|---|
| How do I fit this model? | Is this the right family for the outcome I have? |
| What do the coefficients mean? | Which variables am I entitled to put beside them? |
| Do the assumptions hold? | In what order must I check, given that each failure hides the next? |
| Is the effect significant? | What decision does this number feed, and what does that decision actually need? |
Every question in the right-hand column is settled before the model is fitted or immediately after, and none of them is answered by a fit statistic. That is the uncomfortable common thread: R-squared, p-values and AUC are all silent on every one of these decisions, and a model can score well on all three while answering a question nobody asked.
Decision One: What Kind of Outcome Is This?
Ordinary least squares assumes a continuous outcome with roughly constant variance. Plenty of outcomes are not that, and forcing them into it produces a model that runs without complaint.
| Outcome | Family | What the coefficient means | The trap |
|---|---|---|---|
| Continuous | Gaussian | Change in the outcome per unit | Skew and non-constant variance |
| Continuous, multiplicative | Gaussian on the log | Approximate percentage change | Zeros, and the retransformation bias |
| Binary | Binomial, logit link | Log odds ratio | Reading an odds ratio as a risk ratio |
| Count | Poisson, log link | Log rate ratio | Overdispersion, and the missing offset |
| Count with wide exposure | Poisson with an offset | Log rate ratio | Without the offset you have modeled size |
| Bounded proportion | Beta, or binomial with weights | Depends on the link | Predictions outside [0, 1] |
If the units you are comparing differ in size, a count model without an
offset is a model of size. Districts, stores, machines and policy-years all vary by orders of
magnitude, and log(exposure) entered with its coefficient fixed at 1 is what converts a statement
about counts into a statement about rates. Capstone 28 shows a coefficient that changes sign on this alone.
Decision Two: What Belongs on the Right-Hand Side?
This is the decision with the least statistical content and the largest consequences. Three kinds of variable look identical in a regression table and do completely different things.
The rule is short and it is about time, not statistics. A variable measured before the exposure can be a confounder and is usually safe. A variable measured after it is a mediator or a collider and is usually not. Capstone 25 is built on a control that takes a pay gap from 8.1 percent to 2.3 and deletes the mechanism producing most of it, and the regression output looks identical either way.
Decision Three: In What Order Do You Check?
Regression diagnostics are taught as a checklist and behave as a sequence. Every diagnostic is computed from the residuals, and the residuals are whatever the current model has failed to explain, so each test measures the assumption it names only as far as the other assumptions hold.
| Check | Order | Why it must come there |
|---|---|---|
| Functional form | First | A misspecified mean leaves residuals dominated by systematic error, which makes every later test meaningless |
| Influence | Second | A handful of enormous residuals inflates the overall spread enough to hide a variance pattern in all the rest |
| Constant variance | Third | Only meaningful once the mean is right and no small group dominates the residuals |
| Independent errors | Last | A smooth misfit produces runs of same-signed residuals that are indistinguishable from correlated errors |
Run the four in parallel on a first-draft model and you will find some subset of the faults that are present, and you will not know which you missed. Capstone 26 runs them that way deliberately: three violations are present, and the parallel pass reports one.
Decision Four: What Is the Number For?
The same fitted model supports three quite different products, and each one is judged by something the others do not need.
| The product | What matters | What does not settle it |
|---|---|---|
| An effect estimate | The coefficient, its interval, and what was adjusted for | Predictive accuracy |
| A prediction | Out-of-sample error, and an honest prediction interval | The significance of any coefficient |
| A decision rule | Calibration, the threshold, and the cost ratio behind it | Discrimination, which cannot see the level at all |
The third row is the one that catches people. A model that ranks cases correctly can be wrong about the level of risk by a factor of two, and AUC is mathematically incapable of detecting it, because it reads only the order. Capstone 27 puts two models with the same AUC to four decimal places against a real clinical threshold and gets 951 extra admissions out of one of them.
Youden's index, the elbow of the ROC curve and "0.5" all encode a statement about how much worse one error is than the other. None of them asked anybody what that ratio should be. If a missed case is forty times worse than a false alarm, the threshold is 1/(1+40), and that arithmetic belongs to whoever owns the consequence rather than to the analyst.
What the Fit Statistics Cannot Tell You
Worth stating plainly, because it is the reason all four of these decisions have to be made deliberately rather than discovered.
| The decision | Goes wrong silently as | Detected by |
|---|---|---|
| Wrong outcome family | A model that runs, fits and reports significance | A dispersion test, or predictions outside the possible range |
| A mediator on the right-hand side | A smaller coefficient and a much better R² | Asking whether the control is assigned after the exposure |
| Diagnostics in the wrong order | Three of four tests passing | Re-running them after every repair |
| Wrong product | An excellent AUC on a rule that misprices the decision | Calibration, and net benefit against doing nothing |
In each row the middle column is the danger. None of these failures announces itself, and two of them make the output look better: adjusting for a mediator raises R-squared, and a miscalibrated score can look more cautious than a correct one.
Regression in Data Science & AI
| Where the same decision appears | In its modern costume |
|---|---|
| Choosing a loss function | Squared error, log loss and Poisson deviance are the outcome-family decision under another name |
| Automated feature selection | Makes the mediator and collider decision silently, at scale, with no analyst to argue with |
| Residual analysis of any model | Plotting residuals against each feature finds the one that needs a transform, whatever produced them |
| Probability outputs | Tree ensembles and neural networks are miscalibrated by default while reporting excellent discrimination |
| Explainability | SHAP values inherit the specification: a mediator in the model shows up as an important feature that is stealing the effect |
Part XXXI takes up predictive modeling on its own terms, where the product is a prediction rather than a coefficient and the workflow changes accordingly. The four decisions here do not go away; they change costume.
The Four Projects Ahead
Four projects, each built on one of the four decisions, and each using a dataset where the naive answer is confidently wrong. The Contents always shows what is live.
🎓 Key Takeaways
- ✓A coefficient answers a question four earlier decisions already asked. Change one and the same data gives a different number, with no warning in the output.
- ✓Match the family to the outcome, and remember the offset. A count model without exposure is a model of size.
- ✓Time settles the right-hand side. Measured before the exposure it can be a confounder; measured after it, it is a mediator or a collider.
- ✓Diagnostics are a sequence. Fix the mean, then influence, then variance, then dependence, re-running after each repair.
- ✓Name the product before choosing the metric. An effect estimate, a prediction and a decision rule are judged by three different things.