←Back to the bookChapter 126 · Take it further

Chapter 126 · Take It Further

Operationalizing the Model · Take It Further

Five extensions of the MLOps lifecycle: a KS-test drift check, label-free prediction-drift monitoring, choosing a retraining cadence, a rollback drill that catches a bad challenger, and a shadow evaluation that shows exactly when to promote, each with a picture and a plain-English explanation.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

Operationalizing the Model: Take It Further

Five extensions of the Chapter 126 lifecycle, each with a picture and a plain explanation: a second drift test (the KS test), monitoring the model's own output without labels, choosing a retraining cadence, a rollback drill that catches a bad challenger, and a shadow evaluation that tells you exactly when a challenger is ready to promote. We start by rebuilding the champion and the monitoring log.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

import joblib, warnings; warnings.filterwarnings('ignore')
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from scipy.stats import ks_2samp

RECAP · REBUILD

The champion and the log

Everything from Chapter 126 in a few cells.

try: df = pd.read_csv('../../data/model_monitoring.csv')
except FileNotFoundError: df = pd.read_csv(BASE + 'model_monitoring.csv')
df = df.drop_duplicates('customer_id').reset_index(drop=True)
FEATS = ['tenure_months','monthly_usage_gb','support_tickets','add_ons','monthly_charges','price_increase']
ref = df[df.month==0]
def fit(data): return Pipeline([('i',SimpleImputer(strategy='median')),('s',StandardScaler()),('c',LogisticRegression(max_iter=2000,class_weight='balanced'))]).fit(data[FEATS], data.churned)
champion = fit(ref)
df['risk'] = champion.predict_proba(df[FEATS])[:,1]
auc_by_m = {m: roc_auc_score(df[df.month==m].churned, df[df.month==m].risk) for m in range(1,19)}
print('recap: stable AUC %.3f -> late AUC %.3f' % (np.mean([auc_by_m[m] for m in range(1,7)]), np.mean([auc_by_m[m] for m in range(13,19)])))

recap: stable AUC 0.754 -> late AUC 0.528

EXTENSION 1 · A SECOND OPINION ON DRIFT

The KS test

A statistical test, not just an index.

recent = df[df.month.between(13,18)]
print('Kolmogorov-Smirnov test: reference (m0) vs recent (m13-18)')
for f in FEATS:
    s,p = ks_2samp(ref[f].dropna(), recent[f].dropna())
    flag = '  <-- drift (p<0.001)' if p<0.001 else ''
    print(f'  {f:20s} KS={s:.3f}  p={p:.1e}{flag}')
fig,ax=plt.subplots(figsize=(7.6,4.2)); ks=[ks_2samp(ref[f].dropna(),recent[f].dropna()).statistic for f in FEATS]
order=np.argsort(ks); ax.barh([FEATS[i] for i in order],[ks[i] for i in order],color=[RED if ks[i]>0.15 else EM for i in order])
ax.set(xlabel='KS statistic (max gap between the two distributions)', title='KS test agrees with PSI: tickets and price moved most'); plt.tight_layout(); plt.show()

Kolmogorov-Smirnov test: reference (m0) vs recent (m13-18)
  tenure_months        KS=0.014  p=8.5e-01
  monthly_usage_gb     KS=0.168  p=2.4e-47  <-- drift (p<0.001)
  support_tickets      KS=0.497  p=0.0e+00  <-- drift (p<0.001)
  add_ons              KS=0.013  p=9.1e-01
  monthly_charges      KS=0.176  p=6.9e-52  <-- drift (p<0.001)
  price_increase       KS=0.443  p=2.2e-321  <-- drift (p<0.001)

PSI is not the only drift detector. The Kolmogorov-Smirnov test compares two distributions and returns a p-value; here it flags the same features PSI did, with vanishingly small p-values. Use PSI for a quick dashboard number and KS when you want a formal test. Agreement between two methods is reassuring.

EXTENSION 2 · WATCH THE MODEL'S OWN OUTPUT

Prediction drift, no labels needed

The score distribution shifts before you can score performance.

def psi(a,b,bins=10):
    a=pd.Series(a).dropna(); b=pd.Series(b).dropna(); e=np.quantile(a,np.linspace(0,1,bins+1)); e[0]=-np.inf; e[-1]=np.inf
    r=np.clip(np.histogram(a,e)[0]/len(a),1e-4,None); c=np.clip(np.histogram(b,e)[0]/len(b),1e-4,None); return float(np.sum((c-r)*np.log(c/r)))
base_scores = df[df.month.between(1,3)].risk
pred_psi = [psi(base_scores, df[df.month==m].risk) for m in range(1,19)]
fig,ax=plt.subplots(figsize=(9,4.2))
ax.bar(range(1,19), pred_psi, color=PUR, alpha=0.7); ax.axhline(0.25,color=RED,ls='--',label='drift threshold')
ax.set(xlabel='month', ylabel='PSI of the model output', title='Prediction drift: the score distribution moves on its own'); ax.legend(); plt.tight_layout(); plt.show()
print('the model started predicting churn far more often (tickets rose) -- a label-free warning that something changed')

the model started predicting churn far more often (tickets rose) -- a label-free warning that something changed

You can monitor the model's own predictions. Even with zero outcomes, the distribution of scores the model emits is observable, and here it drifts upward (the model sees more tickets, so it cries churn more often). Prediction drift is a second label-free tripwire, complementing input drift.

EXTENSION 3 · HOW OFTEN TO RETRAIN?

Never, monthly, or on-trigger

Triggered retraining gets most of the benefit for a fraction of the cost.

def rolling_auc(policy):
    aucs=[]; retrains=0; model=champion; last_train=0
    for m in range(7,19):
        cur=df[df.month==m]; aucs.append(roc_auc_score(cur.churned, model.predict_proba(cur[FEATS])[:,1]))
        due = (policy=='monthly') or (policy=='triggered' and psi(ref.support_tickets, cur.support_tickets)>0.25 and m-last_train>=3)
        if due and m<18:
            win=df[df.month.between(m-2,m)]; model=fit(win); retrains+=1; last_train=m
    return aucs, retrains
fig,ax=plt.subplots(figsize=(9,4.4)); mm=list(range(7,19))
for pol,c in [('never',GREY),('monthly',GREEN),('triggered',EM)]:
    a,r=rolling_auc(pol); ax.plot(mm,a,'o-',color=c,lw=2.3,label=f'{pol} ({r} retrains)')
ax.axhline(0.65,color=RED,ls=':'); ax.set(xlabel='month',ylabel='AUC',title='Retraining cadence: triggered nearly matches monthly, far fewer retrains'); ax.legend(); plt.tight_layout(); plt.show()

More retraining is not automatically better. Never retraining lets AUC rot. Retraining every single month keeps it high but is expensive and risks churn from constant model changes. Triggered retraining, fire only when drift crosses the threshold, recovers almost the same performance with a handful of retrains. Cadence is a cost-versus-freshness decision.

EXTENSION 4 · A ROLLBACK DRILL

Catch a bad challenger before it ships

The canary plus a rollback rule is your safety net.

# a DELIBERATELY bad challenger: trained on the OLD regime (months 1-6), useless in the new normal
bad = fit(df[df.month.between(1,6)])
canary = df[df.month==14].copy(); rng=np.random.default_rng(1)
canary['arm']=np.where(rng.random(len(canary))<0.2,'challenger','champion')
auc_champ = roc_auc_score(canary[canary.arm=='champion'].churned, champion.predict_proba(canary[canary.arm=='champion'][FEATS])[:,1])
auc_bad   = roc_auc_score(canary[canary.arm=='challenger'].churned, bad.predict_proba(canary[canary.arm=='challenger'][FEATS])[:,1])
print(f'canary champion arm AUC {auc_champ:.3f} | bad-challenger arm AUC {auc_bad:.3f}')
decision = 'ROLL BACK to champion' if auc_bad < auc_champ - 0.02 else 'promote challenger'
print('rollback rule (challenger must beat champion by 0.02):', decision)
fig,ax=plt.subplots(figsize=(6,4)); ax.bar(['champion','bad\nchallenger'],[auc_champ,auc_bad],color=[GREY,RED])
ax.axhline(auc_champ-0.02,color=RED,ls='--',label='promote line'); ax.set(ylabel='canary AUC',ylim=(0,0.8),title='The canary catches the bad model; rollback fires'); ax.legend(); plt.tight_layout(); plt.show()

canary champion arm AUC 0.511 | bad-challenger arm AUC 0.455
rollback rule (challenger must beat champion by 0.02): ROLL BACK to champion

Not every challenger is an improvement. Here a challenger trained on the stale old regime is no better than the aged champion. Because we shipped it as a small canary and armed a rollback rule, the failure is caught on a slice of traffic and auto-reverted, never reaching all customers. This drill is why gradual rollout exists.

EXTENSION 5 · WHEN IS IT SAFE TO PROMOTE?

Shadow evaluation

Score both models silently and watch for the crossover.

good = fit(df[df.month.between(10,12)])   # the real challenger
rows=[]
for m in range(10,19):
    cur=df[df.month==m]
    rows.append({'month':m,'champion':roc_auc_score(cur.churned,champion.predict_proba(cur[FEATS])[:,1]),
                 'challenger':roc_auc_score(cur.churned,good.predict_proba(cur[FEATS])[:,1])})
shadow=pd.DataFrame(rows)
fig,ax=plt.subplots(figsize=(9,4.4)); ax.plot(shadow.month,shadow.champion,'o-',color=GREY,lw=2.3,label='champion (live)')
ax.plot(shadow.month,shadow.challenger,'o-',color=EM,lw=2.5,label='challenger (shadow)')
cross=int(shadow[shadow.challenger>shadow.champion].month.min()); ax.axvline(cross,color=GREEN,ls='--',label=f'safe to promote (month {cross})')
ax.set(xlabel='month',ylabel='AUC',title='Shadow evaluation: the challenger overtakes, then you promote'); ax.legend(); plt.tight_layout(); plt.show()
print(f'the challenger runs in shadow (scoring, not acting) and beats the champion from month {cross} on -> promote with evidence, not hope')

the challenger runs in shadow (scoring, not acting) and beats the champion from month 10 on -> promote with evidence, not hope

Promote on evidence, not a hunch. In shadow mode the challenger scores live traffic without acting, so you can compare it to the champion on identical, real data month after month. Once it convincingly leads, you promote. This turns a risky guess into a monitored, reversible decision.

Take-it-further summary, in plain terms

• Cross-check drift with a second method: the KS test agrees with PSI.
• Monitor the model's own output: prediction drift warns you with no labels at all.
• Pick a retraining cadence: triggered retraining nearly matches monthly at a fraction of the cost.
• Always keep a rollback armed: a canary catches a bad challenger before it reaches everyone.
• Promote from shadow evidence: run the challenger silently until it clearly wins.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 126 · Take it further