Operationalizing a Forecast: Take It Further¶
Five extensions of the Chapter 140 production pipeline, each hardening the operations: a change-point detector that flags the shift automatically, monitoring interval coverage as a second drift signal, choosing the retraining window (expanding vs rolling), comparing a retraining cadence, and confirming the whole thing with a rolling-origin backtest before deploy. We start by rebuilding the pipeline.
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#0284c7"; DEEP="#075985"; LIGHT="#bae6fd"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#0284c7"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
from statsmodels.tsa.holtwinters import ExponentialSmoothing
import warnings; warnings.filterwarnings('ignore')
def ape(a,f): return np.abs((a-f)/a)*100
try: raw = pd.read_excel('../../data/case-study-operationalizing-a-forecast--demand_ops.xlsx', sheet_name='Data')
except FileNotFoundError: raw = pd.read_excel(BASE + 'case-study-operationalizing-a-forecast--demand_ops.xlsx', sheet_name='Data')
raw['month']=pd.to_datetime(raw['month']); s = raw.set_index('month')['demand'].asfreq('MS'); DEPLOY=48
def refit_forecast(train, h=1): return ExponentialSmoothing(train, trend='add', seasonal='add', seasonal_periods=12).fit()
sched=[float(refit_forecast(s[:m]).forecast(1).iloc[0]) for m in range(DEPLOY,len(s))]
champ=refit_forecast(s[:DEPLOY]).forecast(len(s)-DEPLOY).values
log=pd.DataFrame({'actual':s.values[DEPLOY:],'champ':champ,'sched':np.array(sched)}, index=s.index[DEPLOY:])
print('rebuilt: champion (set-and-forget) and scheduled (monthly refit) forecasts')
rebuilt: champion (set-and-forget) and scheduled (monthly refit) forecasts
err = (log['actual'] - log['champ']).values
mu, sd = err[:12].mean(), err[:12].std() # baseline from the calm start
cusum = np.maximum.accumulate(np.zeros(len(err)))
S=0; flag=None
for i,e in enumerate(err):
S = max(0, S + (e - mu) - 0.5*sd) # one-sided CUSUM for an upward shift
if S > 5*sd and flag is None: flag = log.index[i]
print('CUSUM flags a sustained shift in the errors at:', flag.date() if flag is not None else 'none')
print('a change-point detector turns "someone noticed the chart" into an automatic alarm')
CUSUM flags a sustained shift in the errors at: 2021-07-01 a change-point detector turns "someone noticed the chart" into an automatic alarm
Eyeballing the monitoring chart does not scale. A CUSUM (cumulative sum) detector accumulates the forecast errors and fires when they drift persistently in one direction, catching the regime shift automatically. Wiring a change-point test into the monitor turns drift detection from a human chore into an alarm, the same idea as the PSI trigger in Chapter 130.
hits=tot=0
for m in range(DEPLOY, len(s)):
f=refit_forecast(s[:m]); row=f.simulate(1, repetitions=500).iloc[0] # 500 sims of the 1 forecast month
lo,hi=row.quantile(0.025), row.quantile(0.975); a=s.iloc[m]
hits+=int(lo<=a<=hi); tot+=1
print('scheduled 95%% interval coverage: %d of %d = %.0f%%' % (hits, tot, 100*hits/tot))
print('a run of actuals falling OUTSIDE the interval is an early drift warning, even before MAPE moves')
scheduled 95% interval coverage: 45 of 48 = 94% a run of actuals falling OUTSIDE the interval is an early drift warning, even before MAPE moves
A prediction interval is also a monitor: if actuals keep landing outside the 95% band, the model is losing calibration. Tracking interval coverage (and especially a run of misses) is an early drift signal that can fire before the rolling MAPE has fully deteriorated, useful when you want to react fast.
def sched_window(kind, win=24):
out=[]
for m in range(DEPLOY, len(s)):
tr = s[:m] if kind=='expanding' else s[max(0,m-win):m]
out.append(float(refit_forecast(tr).forecast(1).iloc[0]))
return pd.Series(ape(s.values[DEPLOY:], np.array(out)), index=s.index[DEPLOY:])
for kind in ['expanding','rolling']:
a=sched_window(kind)
print('%-10s window: recovery (3 mo after shift) %.1f%% | full post-shift %.1f%%' % (kind, a['2021-08':'2021-10'].mean(), a['2021-08':].mean()))
print('trade-off: ROLLING adapts faster right after the break; EXPANDING is steadier once the shift is absorbed')
expanding window: recovery (3 mo after shift) 11.1% | full post-shift 3.3%
rolling window: recovery (3 mo after shift) 8.3% | full post-shift 4.5% trade-off: ROLLING adapts faster right after the break; EXPANDING is steadier once the shift is absorbed
The retraining window is a real trade-off. A rolling window of recent months drops stale pre-shift data, so it re-locks onto the new regime faster (lower error in the three months right after the break). An expanding window keeps all history, so it is noisier during the transition but steadier long-run once the shift is absorbed. There is no universal winner: use a rolling window when breaks are frequent, an expanding one when the process is mostly stable, matching the “retrain on the new steady state” lesson from Chapter 130.
FLOOR = 10.0 # the contract's accuracy floor, in percent
def cadence(every, trigger=False):
out=[]; model=refit_forecast(s[:DEPLOY]); last=DEPLOY; refits=0; prev=0.0
for m in range(DEPLOY, len(s)):
alarm = trigger and prev > FLOOR and m > last # last month's error broke the floor
if m-last>=every or alarm: model=refit_forecast(s[:m]); last=m; refits+=1
f = float(model.forecast(m-last+1).iloc[m-last]); out.append(f); prev = ape(s.iloc[m], f)
return ape(s.values[DEPLOY:], np.array(out))[s.index[DEPLOY:]>='2021-08'].mean(), refits
for every in [1,3,6,12]: print('refit every %2d months: post-shift MAPE %.1f%% (%2d refits)' % ((every,)+cadence(every)))
for every in [6,12]: print('every %2d months + on an alarm: post-shift MAPE %.1f%% (%2d refits)' % ((every,)+cadence(every, trigger=True)))
refit every 1 months: post-shift MAPE 3.3% (47 refits)
refit every 3 months: post-shift MAPE 4.3% (15 refits)
refit every 6 months: post-shift MAPE 7.3% ( 7 refits)
refit every 12 months: post-shift MAPE 8.2% ( 3 refits)
every 6 months + on an alarm: post-shift MAPE 3.2% ( 9 refits)
every 12 months + on an alarm: post-shift MAPE 4.1% ( 5 refits)
Refit cadence trades cost against freshness. Of the calendar policies, refitting every month recovers fastest (3.3% after the shift, with 47 refits), and a slow annual cadence leaves the forecast stale for months (8.2%). Triggered retraining refits on the calendar and also as soon as a month's error breaks the 10% floor. A six-month calendar with that trigger reaches 3.2% with only 9 refits, and even the annual calendar drops to 4.1% with 5, because a sudden regime change no longer waits for the next calendar slot.
errs=[]
for m in range(DEPLOY, len(s)):
fc=refit_forecast(s[:m]).forecast(1).iloc[0]; errs.append(ape(s.iloc[m], fc))
errs=np.array(errs)
print('scheduled pipeline rolling-origin MAPE over %d months: mean %.1f%%, worst %.0f%% (the shift month)' % (len(errs), errs.mean(), errs.max()))
print('a full replay of the pipeline on history is the CI test you run BEFORE shipping a change to it')
scheduled pipeline rolling-origin MAPE over 48 months: mean 3.6%, worst 23% (the shift month) a full replay of the pipeline on history is the CI test you run BEFORE shipping a change to it
Before shipping any change to the pipeline, replay it over all of history with a rolling-origin backtest: refit and one-step-forecast at every month, exactly as production would. Its average MAPE (about 3.6%, with spikes near 23% and 22% in the two months the shift hit) is the pipeline's honest track record, and rerunning it after any code change is the continuous-integration test for a forecasting service.
Take-it-further summary, in plain terms¶
- Automate drift detection with a change-point (CUSUM) test, do not rely on a human watching a chart.
- Monitor interval coverage too: a run of actuals outside the band is an early warning.
- Pick the retraining window: a rolling (recent-only) window recovers faster after a regime shift.
- Choose a cadence: a six-month calendar plus an alarm trigger matched monthly refits with 9 refits in place of 47.
- Backtest the whole pipeline with rolling-origin as the CI test before you deploy a change.