Anyone can call .fit(). The hard part, the part that separates a real model from a
self-deluding one, is the discipline around it: how you split the data, how you estimate
performance, how you tune, and how you keep information from leaking from the answer into the inputs. Get
the workflow right and your reported accuracy means something; get it wrong and it is a mirage.
Judge every model on data it has never seen: hold out a test set first, use cross-validation for all decisions, keep preprocessing inside a pipeline, ban any feature you would not know at prediction time, and unlock the test set exactly once.
The Workflow at a Glance
Every serious project follows the same skeleton. The data is split immediately; all the learning and tuning happen on the training portion; the test set stays locked in a vault until the final line.
The single most important habit is to split before you explore. Any scaling, imputing, or feature-picking done on the whole dataset lets the test set influence the model, so those steps live inside the pipeline after the split. The one thing that comes first, on all the rows, is the deterministic cleaning covered in the next section.
Data Cleaning Comes First
Notice the amber box at the front of the diagram. Before a single model is fit, the raw data has to be made trustworthy, and this step is not optional in machine learning; if anything it matters more than in classical statistics, because a model will faithfully learn whatever flaws you leave in. Garbage in, garbage out.
One distinction keeps this consistent with the leakage rule later. Row-level cleaning, fixing typos, parsing dates, correcting data types, and dropping exact duplicates or corrupt rows, uses no information across rows, so it is safe to do on all the data before the split (the amber box). Statistical preprocessing, imputing a missing value with the column mean, scaling, or selecting features, does learn from the data, so it belongs inside the pipeline after the split, exactly where Section 5 puts it to prevent leakage.
| Cleaning issue | Why it bites supervised learning | Why it bites unsupervised learning |
|---|---|---|
| Missing values | most estimators error out on NaN; impute or drop | clustering and PCA cannot run on NaN at all |
| Unscaled features | KNN, SVM, logistic are dominated by the big-scale column | a high-variance feature hijacks the clusters / first component |
| Outliers & bad rows | a few corrupt rows can pull the fit off course | one extreme point drags a centroid or tilts an axis |
| Categories as text | must be encoded to numbers before fitting | same, and the encoding shapes every distance |
| Junk / duplicate columns | add noise and multicollinearity | directly distort the structure that is discovered |
Why cleaning matters for supervised learning. A supervised model is only as good as its features and its labels. Missing values stop most estimators outright, unscaled features distort every distance-based model, and outliers or a handful of corrupt rows can pull the fit off course. The most dangerous case is a noisy or mislabeled target: the model will dutifully learn the wrong answer and then report it with confidence. Clean features and trustworthy labels are the price of admission.
Why cleaning matters for unsupervised learning. Here it matters even more, because there is no label to keep the algorithm honest. K-means and PCA are driven entirely by variance and distance, so an unscaled or outlier-laden feature quietly dominates the clusters or the principal components, and irrelevant columns, with no target to down-weight them, warp the result directly. Worse, an unsupervised method never fails loudly: it always hands back some clusters or components, and it is on you to notice they are meaningless because the input was dirty. Standardize and clean first, then look for structure.
The Split and the Overfitting Trap
Why hold out data at all? Because a flexible model can score 100% on the data it trained on by simply memorizing it, while performing poorly on anything new. Training accuracy is not evidence of learning; the gap between training and validation is.
The notebook draws exactly this curve with validation_curve. As the tree deepens, training accuracy
marches to 1.00 while cross-validated accuracy tops out around a depth of 4 and then declines,
the model has started memorizing noise. The best model sits at the peak of the validation curve, and the widening
gap between the two lines is the definition of overfitting. A held-out set is the only way
to see that gap.
Cross-Validation & Hyperparameter Tuning
A single train/validation split is noisy, one unlucky partition can flatter or damn a model. k-fold cross-validation fixes that by rotating the validation block through the data, so every row is validated exactly once and the scores are averaged.
With a stable estimate in hand, hyperparameter tuning becomes safe. In the notebook,
GridSearchCV cross-validates every combination of tree depth and leaf size and reports the best,
a cross-validated accuracy of 0.74, entirely without touching the test set. When the grid is too
large, RandomizedSearchCV samples it instead. The golden rule holds throughout: tuning
is a decision, and all decisions use cross-validation on the training data, never the test set.
Real-World Example: Pipelines and the Leakage Trap
A lender has 1,000 past loans and wants to predict default. The workflow runs smoothly, until a single leaked feature makes the model look brilliant and useless at the same time. This is the most common, most costly mistake in applied machine learning.
One row per loan with income_k, loan_amount_k,
credit_score, debt_to_income, prior_defaults, late_payments_12m,
and more, plus the label default. A deliberate trap column, sent_to_collections, is
recorded after the outcome.
Wrapping the scaler and model in a Pipeline stops preprocessing leakage: the scaler is
refit inside each cross-validation fold, so the validation rows never influence the training statistics. Then comes
the dramatic one, target leakage:
Adding sent_to_collections rockets cross-validated accuracy from 0.71 to 0.95,
because that column is almost a copy of the label. But it only exists after a loan has defaulted, so
at scoring time (a brand-new application) it is unavailable and the model collapses. The rule that catches every
such trap: a feature must be knowable at the moment you make the prediction. If it encodes the
future, it is leakage.
With leakage banned and the pipeline tuned, the notebook unlocks the test set once: test accuracy 0.74, essentially equal to the cross-validated estimate of 0.74. That agreement is the reward for a clean process, the number you report is the number you will actually get.
The Workflow in Production (MLOps)
Every step here has a heavier-duty counterpart when a model leaves the notebook and runs a business. The discipline does not change, it scales.
| Workflow step (this chapter) | In production it becomes |
|---|---|
| Train / test split | Offline training plus a live holdout or shadow deployment |
| Cross-validation | Time-based validation and backtesting on rolling windows |
| Hyperparameter tuning | Automated sweeps and AutoML with tracked experiments |
| Pipeline | A single deployable artifact (preprocessing + model) |
| Leakage checks | A feature store with point-in-time correctness |
| Evaluate once | Continuous monitoring, drift detection, scheduled retraining |
Improper validation and data leakage are the leading reasons published machine-learning results fail to reproduce, a whole literature documents models that scored beautifully in a paper and fell apart in the wild because a feature leaked the label or the test set was reused during tuning. The workflow in this chapter, split first, cross-validate every decision, pipeline your preprocessing, ban future-knowing features, and touch the test set once, is exactly the checklist that separates a trustworthy result from a headline that does not survive contact with reality.
Run the whole workflow in Python
The companion notebook splits the loan data, draws the overfitting validation curve, cross-validates, tunes with GridSearchCV (with a heatmap of the grid), builds a leakage-safe Pipeline, demonstrates target leakage live, routes mixed column types with a ColumnTransformer (one-hot encoding a categorical feature) into a modern HistGradientBoosting model (the scikit-learn cousin of XGBoost / LightGBM / CatBoost), and evaluates once on the locked test set, each step spelled out.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓Split first: hold out a stratified test set before any exploring or modeling, and unlock it only once.
- ✓Training accuracy lies: a model can memorize the training set; the train-versus-validation gap is overfitting.
- ✓Cross-validation gives a stable estimate for every decision; GridSearchCV tunes hyperparameters on it.
- ✓Pipelines prevent preprocessing leakage by refitting each step inside every fold.
- ✓Target leakage (a feature knowable only after the outcome) inflates scores then fails live; every feature must be knowable at prediction time.
Practice Challenges
Five short challenges on the loan table. Try them with scikit-learn before checking the solutions.
Split first
Hold out 30% as a stratified test set; confirm the default rate matches in both parts.
train_test_split(..., stratify=y).The overfitting gap
Grow a full-depth tree; compare its training accuracy to its cross-validated accuracy.
.score on train vs cross_val_score.Cross-validate
Report the 5-fold CV mean and standard deviation for a depth-4 tree.
cross_val_score(..., cv=5).Tune with GridSearchCV
Search max_depth and min_samples_leaf; report the best settings and CV score.
GridSearchCV(estimator, param_grid, cv=5).Catch the leak
Add sent_to_collections and show cross-validated accuracy jumps unrealistically.
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on the ML workflow. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
With a sound process in place, we can open the models themselves. Distance Metrics covers the rulers, Euclidean, Manhattan, cosine, and more, that so many algorithms use to decide which points count as "close", the foundation of nearest neighbors, clustering, and similarity search.