Most of this book has been about building a good model. This chapter is about the far longer part of its life: everything after training. A deployed model is a piece of software with a peculiar weakness, it was fitted to a snapshot of the world, and the world does not hold still. MLOps is the set of practices that turns a one-off model into a dependable service: reproducible packaging, monitored serving, and a loop that notices decay and responds to it.
Training runs once, top to bottom, and ends with a saved model. Production never ends: the model serves, the inputs it sees slowly change, its accuracy erodes, and something has to catch that and trigger a retrain. The single most important shift in mindset is from build it and ship it to ship it and keep watching it. A model is not a deliverable you finish; it is a service you operate.
The Gap Between a Notebook and a Service
The phrase every team learns to dread is “but it worked in my notebook.” The gap between a model that scores well in development and one that behaves in production is wide, and it is made of a handful of specific, recurring failures.
| The failure | What goes wrong, and what closes the gap |
|---|---|
| Train/serve skew | Development scales or encodes a feature one way; the serving code does it slightly differently. The model sees inputs it never trained on and quietly degrades. Closed by shipping preprocessing and model as one artifact. |
| Irreproducibility | “Which data, which code, which random seed produced this model?” If you cannot answer, you cannot debug it or roll it back. Closed by versioning data, code, and the artifact together. |
| Unvalidated inputs | A model returns a confident number for a negative price or a missing field. Closed by a schema and range check at the serving boundary, before the model is called. |
| Silent drift | The world shifts, the model keeps answering, and no one notices until a business metric falls. Closed by monitoring inputs and performance continuously. |
| No path to update | Decay is found but there is no safe, repeatable way to retrain and redeploy. Closed by an automated pipeline with a champion/challenger check before promotion. |
Notice that none of these is about the model being wrong. Each is about the machinery around the model. That is the whole subject: a mediocre model that is well operated will out-serve a brilliant model that is shipped and forgotten.
Package and Version: The Reproducible Artifact
The first job is to make the model something you can ship, reload, and trust to behave. Two ideas do most of the work.
Ship preprocessing with the model. If your model expects standardized features, the standardizer is part of the model, not a separate step to reimplement in the serving code. Bundle every transform and the estimator into a single pipeline, fit it once, and serialize the whole object. Now serving loads one artifact and hands it raw input; there is no second code path to drift out of sync. In the notebook, the reloaded pipeline scores raw week-7 orders identically to the original, which is the definition of no train/serve skew.
Version everything that produced it. A model is a function of its code, its data, and its configuration. To reproduce or roll back a model you need all three pinned together. A model registry records that: a name and version, a hash of the exact artifact bytes, the training window, the feature list, and the validation metrics. In the notebook the registered record stamps the v1 model as trained on weeks 1 to 6 with a validation accuracy of 0.727 and a Brier score of 0.192, so there is never a question of which model is live or how it scored.
Every practice in this chapter assumes you can answer “what exactly is deployed?” Pin your library
versions, set random seeds, snapshot the training data, and record it all with the artifact. The
later Reproducibility & Version Control chapter goes deeper on
the tooling; here it is enough to see that a registry entry, not a file called model_final_v3.pkl, is
what makes a model operable.
Serve, then Watch: Deployment and Drift
A packaged model has to be reachable and, once it is, watched. Serving comes in two shapes. Batch scoring runs on a schedule over many rows at once, which suits overnight risk scores or weekly churn lists. Online serving answers one request at a time behind an API, which suits a fraud check at checkout. Either way, the serving boundary is where you validate inputs: reject a missing column or an out-of-range value before the model is ever called, because a model will return a confident, meaningless answer for nonsense input rather than an error.
Then comes the practice that separates a maintained model from an abandoned one: drift monitoring. There are two distinct things to watch, and confusing them is a classic mistake.
| Kind of drift | What changes | How you catch it |
|---|---|---|
| Data drift | The input distribution moves. Yesterday's customers are not today's. The relationship the model learned may still hold. | No labels needed. Compare incoming feature distributions to the training baseline, e.g. with the Population Stability Index. Visible immediately. |
| Concept drift | The relationship itself changes. The same inputs now lead to a different outcome, so the learned mapping is stale. | Needs labels. Only measurable once true outcomes arrive, by watching performance fall. Always lags the event. |
The reason data drift matters so much is timing. Labels are slow: a “was this delivery late?” answer arrives hours later, a loan default answer arrives in years. But the inputs are visible the instant a request lands. Watching the inputs gives you an early warning that needs no labels at all, which is often the only warning you get before the damage is done.
When the world shifts, different metrics can move in different directions and mislead you. In the case below, the model's raw recall went up after the shift, which looks like good news, purely because the event it predicts became more common. Meanwhile accuracy fell and the probabilities became miscalibrated. Watch a proper score like the Brier score, which is not flattered by a change in the base rate, alongside input drift. One number is never the whole picture.
Real-World Example: A Model That Decays, and the Retrain That Saves It
A food-delivery company deploys a model that predicts whether an order will arrive late, so it can warn the customer and nudge the courier. It works well at launch. Then, at week 8, the company expands into the suburbs, and the model begins to fail, quietly at first. This is the full production loop in one story.
Delivery orders, 8,400 across 12 weeks. Each row is one order.
week is 1 to 12; the model is trained on the early weeks. distance_km is the delivery
distance, prep_time_min the kitchen preparation time, and courier_load how many orders the
assigned courier already had in flight, a congestion proxy. order_hour is the hour placed,
is_weekend flags weekends, and basket_size is the order value in dollars. The target
late is 1 if the order arrived late. From week 8, a suburban expansion pushes distances and courier
loads up, and worse traffic makes each kilometer run later, so both the inputs and the input-to-lateness
relationship shift at once.
The model is a logistic-regression pipeline trained on weeks 1 to 6 and validated on week 7, all before the expansion. Then it is frozen and left to serve while the notebook plays out twelve weeks of monitoring. The story unfolds in the order a real on-call engineer would live it.
- ● The inputs raise the alarm first, with no labels. The distance PSI is a quiet 0.021 at week 7, then jumps to 0.477 at week 8, far past the 0.2 significant-drift line, the instant the expansion launches. Nobody has reported a late delivery yet.
- ● The labels confirm the cost. Once outcomes arrive, accuracy slips from 0.727 before the shift toward 0.67, and the Brier score worsens from 0.192 to about 0.20: the probabilities are now miscalibrated.
- ● A single metric would have fooled you. Raw recall actually rose after the shift, because late orders became common, not because the model improved. The proper score and the input drift told the truth.
- ● Retraining closes the loop. Refitting on recent weeks 6 to 9 and testing on the unseen weeks 10 to 12 lifts mean accuracy from 0.689 to 0.735, cuts the Brier score from 0.194 to 0.170, and raises recall from 0.64 to 0.83. The challenger beats the frozen champion on every metric, so it earns promotion to v2.
The shape of this story is the entire point of MLOps. Nothing went wrong with the model; the world moved, and the system was built to notice and respond. The input monitor bought time before the labels arrived, a proper score kept a misleading metric from hiding the damage, and a champion/challenger check made the update safe. That loop, not any one clever model, is what keeps machine learning working in the real world.
MLOps in the Machine Learning & AI Stack
Everything above was done by hand on one machine to make the moving parts visible. In practice a platform automates each step, and a shared vocabulary has grown up around them.
| Practice | What it automates, and where you meet it |
|---|---|
| Experiment tracking | Every training run's parameters, data version, and metrics are logged so results are comparable and reproducible. Tools: MLflow, Weights & Biases. |
| Model registry | A versioned store of artifacts with stages (staging, production, archived) and a promotion history, so you always know what is live and can roll back. |
| Feature store | A shared, versioned source of feature definitions serving training and production alike, so the numbers a model learned from match the ones it later sees. The structural cure for train/serve skew. |
| CI/CD for models | An automated pipeline that tests, validates, and deploys a model on a trigger, with a champion/challenger gate before promotion. The retrain loop, made safe and repeatable. |
| Monitoring & alerting | Continuous drift and performance dashboards that page a human when a metric crosses a threshold, instead of waiting for a business report to reveal the failure. |
| Serving infrastructure | Containerized endpoints that scale with traffic, with A/B or canary rollout so a new model is tested on a slice before it takes all the load. |
| LLMOps | The same loop for large language models, with new instruments: prompt and version management, evaluation harnesses, guardrails, retrieval freshness, and cost and latency budgets. |
A widely cited paper from Google, Hidden Technical Debt in Machine Learning Systems, made the case that the model is a small box in a large diagram: the code that actually runs in production is dominated by data plumbing, configuration, monitoring, and glue. The influential “ML Test Score” rubric that followed turned that observation into a checklist of tests for data, model, and infrastructure. The research consensus is blunt and useful: in a deployed system, the statistics is the easy part, and the engineering around it is where reliability is won or lost.
Companion notebook
The whole production loop on one machine, no platform: package preprocessing and model into one artifact and prove the reloaded copy scores raw input identically, register the version with a content hash and its metrics, serve predictions behind an input-validation gate, detect the week-8 shift from the inputs alone before any labels arrive, confirm the cost with accuracy and a calibration score once labels come back, and retrain to recover, promoting the challenger only after it beats the champion on held-out weeks.
Requires scikit-learn,
joblib, pandas, numpy, matplotlib, and openpyxl. All are
preinstalled on Colab.
🎓 Key Takeaways
- ✓Training is a line; production is a loop: data, train, package, deploy, monitor, and back to data. The loop, not the model alone, is the deliverable.
- ✓Ship preprocessing with the model as one artifact to kill train/serve skew, and version it in a registry so you always know what is deployed.
- ✓Validate inputs at the serving boundary: a model returns a confident answer for nonsense, so reject bad schemas and out-of-range values before scoring.
- ✓Data drift needs no labels and warns early; concept drift needs labels and always lags. Watch the inputs (e.g. PSI) to catch trouble before outcomes arrive.
- ✓No single metric tells the truth under drift: recall rose here while accuracy fell and calibration worsened. Track a proper score like the Brier score alongside drift.
- ✓Close the loop with retraining, and promote a challenger only after it beats the frozen champion on held-out data.
- ✓In production the engineering dwarfs the model: platforms automate packaging, registry, feature stores, CI/CD, and monitoring, but they add no step you have not now seen by hand.
Practice Challenges
Five exercises on the delivery-orders data. Full solutions are in the companion solutions notebook.
Package with no skew
Fit a preprocessing-plus-model pipeline on weeks 1 to 6, serialize it, reload it, and prove the reloaded artifact scores raw input identically.
joblib.dump / joblib.load, then compare predict and predict_proba.Register the model
Build a JSON registry record with a hash of the artifact bytes, the training window, the feature list, and the week-7 metrics.
hashlib.md5(open(path,"rb").read()).Harden the endpoint
Write a serve function that rejects missing columns and out-of-range values, then returns the label and probability for valid input.
predict_proba.Monitor every feature
Compute PSI for all six features at week 10 versus the weeks 1 to 6 baseline, and rank which drifted most.
Trigger and gate a retrain
Retrain on weeks 6 to 9, evaluate champion and challenger on weeks 10 to 12, and only “promote” if the challenger improves accuracy.
Solutions notebook
All five challenges worked in code: a packaged pipeline that reloads with identical predictions, a registry record with a content hash, a hardened serve function that rejects bad input, a per-feature PSI ranking at week 10, and a champion/challenger retrain with an explicit promotion decision.
Quiz: Test Yourself
Eight questions on the production loop: packaging, versioning, serving, drift, monitoring, and retraining. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
This part carried the toolkit past the standard curriculum and into the machinery behind modern practice. Bayesian Inference in Practice replaced a single estimate with a distribution over belief; Causal Inference asked what an intervention would do, not merely what correlates; Survival Analysis handled outcomes that have not happened yet; and Structural Equation Modeling & Mixed Models dealt with nested data and constructs you cannot measure directly. Then Deep Learning, Recommendation Systems, and NLP & Large Language Models built up the models behind modern AI, Big Data & Scaling supplied the engineering to run them at size, and this chapter closed the loop from a trained model to a maintained service. Next, the Tools & Workflow part turns to the practical craft, the languages and software that carry all of this out. Browse the full Contents for what is published and what is on the way.