Every part before this one estimated something. This one predicts something, and that single change moves almost every decision in the framework, including two that stop existing and two that appear from nowhere.
Seven projects across supervised, unsupervised and reinforcement learning. In each one the model trains without complaint, the headline metric looks good, and the naive answer is wrong: by a metric that flatters a useless model, by a feature that could not have existed, by clusters in data with no clusters in it, and by a policy evaluated on the log it came from.
What Changes, and What Does Not
The twelve-step framework from Capstone 1 still runs. Four of its steps mean something different here, and pretending otherwise is how most first machine-learning projects go wrong.
| Framework step | In an inference project | In a prediction project |
|---|---|---|
| State the question | Which effect, on whom, compared with what | Which decision, made how often, at what cost per error |
| Check the assumptions | Normality, constant variance, independence | Validation design: does the split mimic how the model will be used? |
| Choose the method | Follows from the data type and the design | Follows from the data type, and then from a bake-off nobody can skip |
| Report the result | An effect with an interval | Out-of-sample performance, on a metric tied to the cost |
| State the limitations | Confounding, generalizability, measurement | Leakage, drift, and the population the training data represents |
Two things vanish. Statistical significance stops being interesting: nobody asks whether a random forest's improvement over a baseline is significant, they ask whether it is large enough to pay for the pipeline. And the coefficient stops being the product, which is what makes explainability a separate problem rather than a free by-product.
Two things appear. Leakage, which has no analog in the inference chapters, and the baseline, which does: it is the "compared with what" of the first row, and it is skipped just as often.
Validation Design Replaces Assumption Checking
A held-out set is not a formality. It is the only instrument you have, and how you cut it decides what it measures. The rule is short: the split must mimic the way the model will actually be used.
| If the model will | Split by | A random split would |
|---|---|---|
| Score new customers | Customer | Put the same customer in train and test |
| Forecast next month | Time, with a rolling origin | Train on the future to predict the past |
| Score a new hospital's patients | Hospital, grouped | Learn that hospital's quirks and call it generalization |
| Run on next year's transactions | Time, then check drift | Report a number that decays after deployment |
| Score any unit once | At random, stratified on a rare outcome | Be correct, which is why the default is a default |
Train fits the parameters. Validation chooses between models and settings. Test is looked at once, at the end, to report a number. The moment you use the test set to choose anything, it has become a validation set and you no longer have an honest estimate of performance. Cross-validation replaces the first two; it does not replace the third.
Leakage, and Why It Looks Like Success
Leakage is information in the training data that will not exist at the moment of prediction. It is the defining failure of applied machine learning, and its signature is a model that performs implausibly well.
The subtle cases are the ones that survive review. Scaling before splitting lets the test set contribute its mean and standard deviation to the training data. Imputing before splitting does the same with medians. Selecting features on the full dataset and then cross-validating the chosen ones reports the performance of a search that already saw the answers. The fix in all three cases is the same: every step that learns anything from data belongs inside the fold, which is what a pipeline object is for.
The Metric Comes From the Cost
Accuracy is a default that suits almost nothing. The metric is an accounting decision about what being wrong costs, and it should be settled before a model is fitted.
| Situation | Metric | Because |
|---|---|---|
| Rare positive, both errors cost | Precision-recall AUC, then a cost-weighted threshold | ROC-AUC flatters a model when negatives dominate |
| The number itself is used | Calibration, Brier score | Ranking is not enough when a probability is read |
| A ranked shortlist | Precision at k | Only the top of the list will ever be looked at |
| Continuous outcome, outliers matter | MAE, or a quantile loss | Squared error lets a few large errors set the model |
| Continuous outcome, scale varies | MASE or a percentage error, carefully | Percentage errors break at values near zero |
| A policy that acts | Expected reward, evaluated off-policy | The log was produced by a different policy than the one you are testing |
Notice how much of that table is about asymmetry. Most real decisions have two errors that cost different amounts, and almost every default metric assumes they cost the same. Capstone 29 makes this concrete on a fraud problem where the two errors differ by three orders of magnitude.
The Baseline That Has to Be Beaten
A performance number on its own means nothing. It means something against the cheapest thing that could have been done instead, and that comparison is what tells you whether the model earns its maintenance.
| Problem | The baseline to beat |
|---|---|
| Classification | Predict the majority class; then a logistic regression on five obvious features |
| Regression | Predict the mean; then a linear model |
| Forecasting | Predict last period, or the same period last year |
| Ranking or targeting | The current business rule, whatever it is |
| Any of them | Doing nothing, which is free and sometimes wins |
The last row is not a joke. A model that improves accuracy by a point and costs an engineer a week a month to maintain has a negative return, and no metric on the model's own scorecard will surface that.
Explanations, and What They Are Evidence Of
Because the coefficient is no longer the product, explanation becomes a separate exercise with its own tools and its own failure mode.
SHAP and LIME explain what the model did. They do not explain what the world does. A feature with a large SHAP value is a feature the model leaned on, which may be because it causes the outcome, because it proxies for something that does, or because it leaked. An explanation is evidence about the model and is not evidence about the mechanism, and the difference matters most in exactly the settings where somebody wants to act on it.
That said, they are the best leakage detector there is. A feature at the top of an importance ranking that nobody expected to matter is the single most reliable sign that something in the pipeline knows the answer.
When There Is No Outcome at All
Three of the projects ahead have no labels, which removes the instrument the rest of this chapter is built on. Without a held-out outcome there is nothing to be right or wrong about, and the discipline has to come from somewhere else.
| Without labels you cannot | So instead you |
|---|---|
| Measure accuracy | Measure stability: does the answer survive resampling the data? |
| Tune to a metric | Compare against what pure noise produces, which is never nothing |
| Validate a threshold | Price it: how many alerts per day can a human actually read? |
| Prove the structure is real | State what would have to be true for it to be, and test that |
Clustering always returns clusters and anomaly detection always returns anomalies, on any data whatsoever, including data generated with no structure in it at all. Capstones 33 and 35 run exactly that experiment, because the output looks identical either way.
The Seven Projects Ahead
Seven projects across the three families, each one built around a decision the metric cannot make for you. The Contents always shows what is live.
🎓 Key Takeaways
- ✓Validation design replaces assumption checking. The split has to mimic how the model will be used, or the number it produces is about a situation that will never occur.
- ✓Leakage looks like success. Ask of every feature whether it would exist, with that value, at the moment of prediction, and keep every fitted step inside the fold.
- ✓The metric is an accounting decision. Most real problems have two errors that cost different amounts, and almost every default assumes they do not.
- ✓A number means nothing without a baseline. Including doing nothing, which is free and sometimes wins.
- ✓Explanations describe the model, not the world. Which is also what makes them the best leakage detector available.