Every method in this book so far has quietly assumed that the rows could be shuffled without loss. Draw a random sample, split at random, resample at random. On a time series, every one of those moves is a mistake, and the mistake is invisible in the output.
Three projects on the parts of forecasting that the textbook examples step around: forecasts at several levels of a business that are required to add up, demand that is zero in most periods so that the usual error measures cannot be computed at all, and the backtesting discipline that decides whether any of it was real. In each one the model fits, the accuracy number looks respectable, and the honest answer is somewhere else.
What Time Takes Away
The modeling workflow in Chapter 190 is still the right shape. Four of its moves stop working, and they stop working for the same reason: the rows arrived in an order, and the order carries information the model must not be allowed to see.
| Modeling workflow | What it assumed | What forecasting needs instead |
|---|---|---|
| Random train and test split | Rows are interchangeable | A time split: train on the past, test on the future, never the reverse |
| k-fold cross-validation | Any fold can be held out | Rolling origin: a sequence of forecasts, each made from a later point |
| One accuracy number | All predictions are alike | Accuracy by horizon: one step ahead and twelve steps ahead are different problems |
| Feature engineering | A row's features describe that row | The information set at time t: only what had happened by then |
| A point prediction | The interval is optional | An interval whose coverage is measured, because a plan is built on the range |
One thing does not change, and it is worth saying plainly. The baseline still decides whether anything was achieved. In forecasting the baselines are unusually strong, which is why so many projects that report a respectable error have in fact lost to a rule that fits on one line.
Never Shuffle, and What Shuffling Hides
A random split on a time series does not merely bend a rule. It changes the question. The model is handed January and March and asked about February, which is interpolation, and then judged as though it had been asked about next year, which is extrapolation. Those two tasks have almost nothing in common, and the first is far easier.
A random split on a seasonal series usually produces an excellent score, because the model has seen the same month in adjacent years and only has to remember it. The number is real. It is just an answer to a question nobody asked.
Three leaks are more common than the split itself, and all three survive a careful reading of the code because none of them look like time at all.
| Leak | What it looks like | The fix |
|---|---|---|
| Fitted preprocessing | Scaling, imputation or a seasonal index computed on the whole series before splitting | Fit every step on the training window only, inside the fold |
| A centered window | A rolling mean or median that averages the periods either side of each point | Trailing windows only. A centered average of week 20 contains week 21 |
| An aggregate joined back | A monthly total, a store's annual average, a category share, attached to every row inside that month | Compute it from periods strictly before the row, expanding rather than whole |
The third one is the quiet killer, because it is how most people build a useful feature. A restaurant's average delivery time is a genuinely predictive feature, and computing it over the full history means the feature for January already knows about December of the following year. The version that works uses only what had happened before each row, which is slower to write and is the difference between a model and a memory.
The Origin and the Horizon
Two numbers turn a forecast into something that can be checked, and most reports carry neither. The origin is the last period the model was allowed to see. The horizon is how far past it the forecast reaches. A single accuracy figure with no horizon attached is an average over whatever mixture of easy and hard predictions happened to be in the test set.
Errors grow with the horizon, almost always, and they grow at a rate that is itself worth reporting. A model that is excellent one step out and no better than the naive rule at six steps is telling you exactly how far ahead the business can plan on it. Pooling those into a single figure hides the only part of the answer that affects a decision.
Baselines That Are Harder Than They Look
Forecasting has the strongest naive baselines in applied statistics, and people who have not tried to beat them consistently underestimate how good they are. Each of these fits on one line and none of them has a parameter to tune.
| Baseline | The forecast is | When it is hard to beat |
|---|---|---|
| Naive | The last observed value, repeated | Anything close to a random walk, which includes most prices |
| Seasonal naive | The value from the same period one season ago | Strongly seasonal series with a stable pattern. On monthly retail this is a serious opponent |
| Drift | The last value plus the average change per period so far | Series with a persistent trend and little seasonality |
| Mean | The average of the history | Series with no trend and no season, which is rarer than it sounds |
These are also the denominator of the one error measure built for the job. MASE, the mean absolute scaled error, divides the forecast's mean absolute error by the mean absolute error of the one-step naive forecast on the training data. Below 1 means better than naive, above 1 means worse, and the units cancel, so a value is comparable across series measured in dollars, pallets and page views. That last property is what makes it the right choice when a report covers many series at once.
Every forecast in this part is reported against the seasonal naive rule on the same backtest, at the same horizons, on the same origins. A model that cannot state that comparison has not established anything, however sophisticated it is.
The Metric Follows From the Decision
The default in most forecasting software is a percentage error, and it is the wrong default more often than any other choice in this part. Two of its problems are arithmetic and one is a matter of incentives.
| Measure | What it aims at | Where it fails |
|---|---|---|
| MAPE | Average percentage error | Undefined when the actual is zero, and unbounded above while capped at 100 percent below, so it quietly rewards under-forecasting |
| sMAPE | A symmetric repair of MAPE | Bounded, and still unstable near zero and still not symmetric in the way the name suggests |
| RMSE | The conditional mean | Dominated by the largest errors, which is correct only if large errors really do hurt disproportionately |
| MAE | The conditional median | Optimizing it produces a forecast that is under half the time and over half the time, which is not what a stock level wants |
| MASE | Performance relative to naive | Needs a sensible naive rule to scale against, and says nothing about the size of the error in business units |
| Pinball loss | A stated quantile | Requires you to name the quantile, which means naming the cost of running short against the cost of holding stock |
The asymmetry in MAPE deserves the extra sentence. If the actual is 100 and you forecast 200, the percentage error is 100 percent. If you forecast 0, it is also 100 percent, and there is nothing below zero to forecast. Over-forecasting can be penalized without limit and under-forecasting cannot, so a team tuned on MAPE drifts low, and the drift looks like an improving score.
The way out is the same one the modeling workflow used: name the decision first. If the forecast sets a stock level and running out costs three times as much as holding a spare unit, the target is not the middle of the distribution at all, it is the 75th percentile of it, and the measure that scores that honestly is the pinball loss at that quantile. The spare-parts project later in this part is built on exactly that gap between a good average and a workable service level.
A Forecast Without an Interval Is Not a Forecast
Nobody plans against a point. They plan against a range, and they choose the width of the range from what they can afford to be wrong by. That makes the interval the deliverable and the point estimate the summary, which is the reverse of how forecasts are usually presented.
An interval makes a claim that can be checked. Ninety-five percent of future values should land inside a 95 percent interval. Backtesting gives you enough forecasts to count, and the count is nearly always disappointing.
Two habits follow. Report coverage alongside accuracy, every time, from the same backtest that produced the error figures. And when the band is too narrow, say so rather than widening it by hand, because the gap between nominal and empirical coverage is itself information about how much the model does not know.
Forecasting in Data Science & AI
Forecasting is the corner of applied statistics where the classical methods have held up best against machine learning, and the reason is instructive. Most business series are short, noisy and seasonal, which is the regime where a flexible model has the least to work with and the most opportunity to overfit.
| Family | Examples | Where it earns its place |
|---|---|---|
| Classical | Exponential smoothing, ARIMA, seasonal decomposition | Single series, short history, strong seasonality. Fast, interpretable, and a genuine contender |
| Tabular machine learning | Gradient boosting on lag and calendar features | Many related series with covariates. Won the M5 retail competition outright |
| Global deep models | DeepAR, N-BEATS, temporal fusion transformers | Thousands of series that share structure, where one model can learn a pattern no single series shows clearly |
| Pretrained time-series models | Chronos, TimeGPT, Moirai and their successors | Zero-shot forecasts on a new series with no fitting at all, useful as a fast, strong baseline |
| Combination | Averaging or weighting several of the above | Almost always. Combining forecasts is the most reliably useful trick in the field |
The M4 competition in 2018 was won by a hybrid that put exponential smoothing together with a recurrent network, and the runner-up was a weighted combination of statistical methods. The M5 competition in 2020, on hierarchical Walmart sales, was dominated by gradient boosting on engineered features. Two lessons survive both. Combinations beat their own components, reliably enough to be the default rather than a refinement. And the winning margins over a well-implemented seasonal naive rule are smaller than newcomers expect, which is why the baseline is not a formality in this part.
Scale has not dissolved the three problems ahead either. A pretrained model still returns forecasts that do not add up across a hierarchy, still cannot compute a percentage error on a series of zeros, and still produces intervals whose coverage has to be measured rather than trusted.
The Three Projects Ahead
Three projects, each built on a failure that survives a good model. The Contents always shows what is live.
🎓 Key Takeaways
- ✓Never shuffle, and check the three quiet leaks. Fitted preprocessing, centered windows and joined-back aggregates all reach into the future without looking like time at all.
- ✓A forecast has an origin and a horizon. Report accuracy by horizon; a single pooled figure averages over easy and hard predictions and hides how far ahead the business can plan.
- ✓The seasonal naive rule is a serious opponent. State the comparison against it on the same backtest, or you have not established that the model achieved anything.
- ✓MAPE rewards under-forecasting and dies on zeros. Pick the measure from the decision: the mean, the median or a named quantile, and use MASE when the report spans series in different units.
- ✓Coverage is a claim you can count. Measure it on the backtest, expect it to be short of nominal, and report the gap rather than papering over it.