←Back to the bookChapter 135 · Take it further

Chapter 135 · Take It Further

Operationalizing a Forecast · Take It Further

Five ways to harden the forecasting pipeline: automatic change-point detection, interval-coverage monitoring, expanding vs rolling retraining windows, choosing a retraining cadence, and a rolling-origin backtest as the CI test before deploy.

Statistics, Data Science and AI: A Visual Handbook · John Fisher · 2026

Operationalizing a Forecast: Take It Further

Five extensions of the Chapter 135 production pipeline, each hardening the operations: a change-point detector that flags the shift automatically, monitoring interval coverage as a second drift signal, choosing the retraining window (expanding vs rolling), comparing a retraining cadence, and confirming the whole thing with a rolling-origin backtest before deploy. We start by rebuilding the pipeline.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns   # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#0284c7"; DEEP="#075985"; LIGHT="#bae6fd"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#0284c7"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
   "axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
   "axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"

from statsmodels.tsa.holtwinters import ExponentialSmoothing
import warnings; warnings.filterwarnings('ignore')
def ape(a,f): return np.abs((a-f)/a)*100

RECAP · REBUILD

The series and the scheduled forecasts

Everything from Chapter 135 in a few cells.

try: raw = pd.read_excel('../../data/case-study-operationalizing-a-forecast--demand_ops.xlsx', sheet_name='Data')
except FileNotFoundError: raw = pd.read_excel(BASE + 'case-study-operationalizing-a-forecast--demand_ops.xlsx', sheet_name='Data')
raw['month']=pd.to_datetime(raw['month']); s = raw.set_index('month')['demand'].asfreq('MS'); DEPLOY=48
def refit_forecast(train, h=1): return ExponentialSmoothing(train, trend='add', seasonal='add', seasonal_periods=12).fit()
sched=[float(refit_forecast(s[:m]).forecast(1).iloc[0]) for m in range(DEPLOY,len(s))]
champ=refit_forecast(s[:DEPLOY]).forecast(len(s)-DEPLOY).values
log=pd.DataFrame({'actual':s.values[DEPLOY:],'champ':champ,'sched':np.array(sched)}, index=s.index[DEPLOY:])
print('rebuilt: champion (set-and-forget) and scheduled (monthly refit) forecasts')

rebuilt: champion (set-and-forget) and scheduled (monthly refit) forecasts

EXTENSION 1 · AUTOMATIC CHANGE-POINT DETECTION

Flag the shift without human eyes

A CUSUM on the forecast errors.

err = (log['actual'] - log['champ']).values
mu, sd = err[:12].mean(), err[:12].std()          # baseline from the calm start
cusum = np.maximum.accumulate(np.zeros(len(err)))
S=0; flag=None
for i,e in enumerate(err):
    S = max(0, S + (e - mu) - 0.5*sd)              # one-sided CUSUM for an upward shift
    if S > 5*sd and flag is None: flag = log.index[i]
print('CUSUM flags a sustained shift in the errors at:', flag.date() if flag is not None else 'none')
print('a change-point detector turns "someone noticed the chart" into an automatic alarm')

CUSUM flags a sustained shift in the errors at: 2021-07-01
a change-point detector turns "someone noticed the chart" into an automatic alarm

Eyeballing the monitoring chart does not scale. A CUSUM (cumulative sum) detector accumulates the forecast errors and fires when they drift persistently in one direction, catching the regime shift automatically. Wiring a change-point test into the monitor turns drift detection from a human chore into an alarm, the same idea as the PSI trigger in Chapter 126.

EXTENSION 2 · INTERVAL COVERAGE AS A DRIFT SIGNAL

When the actuals leave the band

A second, label-light monitor.

hits=tot=0
for m in range(DEPLOY, len(s)):
    f=refit_forecast(s[:m]); row=f.simulate(1, repetitions=500).iloc[0]   # 500 sims of the 1 forecast month
    lo,hi=row.quantile(0.025), row.quantile(0.975); a=s.iloc[m]
    hits+=int(lo<=a<=hi); tot+=1
print('scheduled 95%% interval coverage: %d of %d = %.0f%%' % (hits, tot, 100*hits/tot))
print('a run of actuals falling OUTSIDE the interval is an early drift warning, even before MAPE moves')

scheduled 95% interval coverage: 45 of 48 = 94%
a run of actuals falling OUTSIDE the interval is an early drift warning, even before MAPE moves

A prediction interval is also a monitor: if actuals keep landing outside the 95% band, the model is losing calibration. Tracking interval coverage (and especially a run of misses) is an early drift signal that can fire before the rolling MAPE has fully deteriorated, useful when you want to react fast.

EXTENSION 3 · EXPANDING vs ROLLING WINDOW

Which retraining window recovers faster?

After a shift, recent data matters more.

def sched_window(kind, win=24):
    out=[]
    for m in range(DEPLOY, len(s)):
        tr = s[:m] if kind=='expanding' else s[max(0,m-win):m]
        out.append(float(refit_forecast(tr).forecast(1).iloc[0]))
    return pd.Series(ape(s.values[DEPLOY:], np.array(out)), index=s.index[DEPLOY:])
for kind in ['expanding','rolling']:
    a=sched_window(kind)
    print('%-10s window: recovery (3 mo after shift) %.1f%%  |  full post-shift %.1f%%' % (kind, a['2021-08':'2021-10'].mean(), a['2021-08':].mean()))
print('trade-off: ROLLING adapts faster right after the break; EXPANDING is steadier once the shift is absorbed')

expanding  window: recovery (3 mo after shift) 11.1%  |  full post-shift 3.3%

rolling    window: recovery (3 mo after shift) 8.3%  |  full post-shift 4.6%
trade-off: ROLLING adapts faster right after the break; EXPANDING is steadier once the shift is absorbed

The retraining window is a real trade-off. A rolling window of recent months drops stale pre-shift data, so it re-locks onto the new regime faster (lower error in the three months right after the break). An expanding window keeps all history, so it is noisier during the transition but steadier long-run once the shift is absorbed. There is no universal winner: use a rolling window when breaks are frequent, an expanding one when the process is mostly stable, matching the “retrain on the new steady state” lesson from Chapter 126.

EXTENSION 4 · RETRAINING CADENCE

How often to refit?

Triggered retraining beats a slow calendar.

def cadence(every):
    out=[]; model=refit_forecast(s[:DEPLOY]); last=DEPLOY
    for m in range(DEPLOY, len(s)):
        if m-last>=every: model=refit_forecast(s[:m]); last=m
        out.append(float(model.forecast(m-last+1).iloc[m-last]))
    return ape(s.values[DEPLOY:], np.array(out))[s.index[DEPLOY:]>='2021-08'].mean()
for every in [1,3,6,12]: print('refit every %2d months: post-shift MAPE %.1f%%' % (every, cadence(every)))
print('more frequent refits recover faster; the best policy is often TRIGGERED (refit when the monitor alarms)')

refit every  1 months: post-shift MAPE 3.3%

refit every  3 months: post-shift MAPE 4.2%

refit every  6 months: post-shift MAPE 7.3%
refit every 12 months: post-shift MAPE 8.2%
more frequent refits recover faster; the best policy is often TRIGGERED (refit when the monitor alarms)

Refit cadence trades cost against freshness. Refitting every month recovers fastest from the shift; a slow annual cadence leaves the forecast stale for many months. In practice the best of both is triggered retraining, refit on the schedule but also immediately whenever the drift monitor alarms, so a sudden regime change does not wait for the next calendar slot.

EXTENSION 5 · BACKTEST THE PIPELINE

Prove it before you deploy

Rolling-origin over the whole history.

errs=[]
for m in range(DEPLOY, len(s)):
    fc=refit_forecast(s[:m]).forecast(1).iloc[0]; errs.append(ape(s.iloc[m], fc))
errs=np.array(errs)
print('scheduled pipeline rolling-origin MAPE over %d months: mean %.1f%%, worst %.0f%% (the shift month)' % (len(errs), errs.mean(), errs.max()))
print('a full replay of the pipeline on history is the CI test you run BEFORE shipping a change to it')

scheduled pipeline rolling-origin MAPE over 48 months: mean 3.6%, worst 23% (the shift month)
a full replay of the pipeline on history is the CI test you run BEFORE shipping a change to it

Before shipping any change to the pipeline, replay it over all of history with a rolling-origin backtest: refit and one-step-forecast at every month, exactly as production would. Its average MAPE (about 3%, with a single spike at the shift) is the pipeline's honest track record, and rerunning it after any code change is the continuous-integration test for a forecasting service.

Take-it-further summary, in plain terms

• Automate drift detection with a change-point (CUSUM) test, do not rely on a human watching a chart.
• Monitor interval coverage too: a run of actuals outside the band is an early warning.
• Pick the retraining window: a rolling (recent-only) window recovers faster after a regime shift.
• Choose a cadence: frequent or, better, triggered retraining recovers fastest.
• Backtest the whole pipeline with rolling-origin as the CI test before you deploy a change.

---

Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher

←Back to the bookChapter 135 · Take it further