Two forecasters show you their numbers. One is accurate on a typical day but blew a single big call; the other is always a little low. Which is better? There is no answer until you say which errors cost you, and that choice is the metric. A forecast metric is a loss function in disguise, so learning the family is really learning to state your priorities precisely.
MAE treats every miss equally; RMSE squares them, so one big miss dominates. On our data one model wins by MAE and the other edges it by RMSE, both true. Report several metrics, check they agree, and pick the one that mirrors the cost of being wrong.
The Metric Family
Every metric starts from the same residual, actual − forecast, and then makes one choice: how to
summarize a whole column of them. Those choices sort into four groups.
The four families answer different questions. Scale-dependent metrics stay in the data's units, great for one series, useless for comparing dollars to units. Percentage metrics free you from the scale but misbehave near zero. Scaled and relative metrics divide your error by a naive benchmark, turning it into a skill score where below 1 means “better than doing nothing.” Pick from the group that matches your question.
ME, MAE & RMSE: How You Punish Big Errors
The three workhorses differ only in how they treat a large miss, and that difference can flip the ranking.
Mean Error (ME) averages the signed residuals, so it measures bias: a systematic lean high or low. It says nothing about accuracy, errors can cancel to near zero while individually being large. MAE averages the absolute errors, giving the typical miss in real units. MSE and its root RMSE square first, so a single large error weighs far more. RMSE is always at least MAE, and the gap widens with the spread. That is why our two models split the decision: model_a is best on MAE but its one missed spike, squared, pushes its RMSE just past model_b's.
Percentage, Scaled & Relative Errors
Units-based metrics cannot compare a demand forecast to a revenue forecast. Three families fix that, each with its own character.
MAPE (mean absolute percentage error) is the most quoted, intuitive and scale-free, but it explodes when actuals approach zero and penalizes over- and under-forecasts asymmetrically. sMAPE symmetrizes the denominator to ease that. The most robust choice is MASE, the MAE divided by the naive forecast's MAE, and Theil's U, the RMSE version: both are skill scores where below 1 beats naive. On our data model_a scores MASE 0.26 and model_b 0.60, so both clearly earn their keep over doing nothing, the very first thing a forecast must prove.
Is the Difference Real? The Diebold-Mariano Test
A leaderboard is only useful if its gaps are trustworthy. A tiny RMSE difference on 24 points might be pure luck.
The Diebold-Mariano (DM) test formalizes the comparison: it takes the two models' per-period losses (say squared errors), forms the difference series, and tests whether its mean is significantly away from zero. On our data, model_a vs model_b gives p ≈ 0.90, the RMSE gap is not statistically distinguishable, it rides entirely on one missed spike, so crowning model_b on RMSE would be overconfident. By contrast, model_a vs naive gives p ≈ 0.0001: that improvement is real. Pair every ranking with a check like this, and confirm the winner's residuals look like white noise (a Ljung-Box test), which means no obvious signal was left on the table.
Compute ME (catch bias), MAE and RMSE (typical vs large-error accuracy), a MASE or Theil's U against naive (skill), and a Diebold-Mariano test between your top two. If the metrics agree, ship the winner; if they disagree, let the cost of errors break the tie.
Real-World Example: Grading Three Forecasts
The companion notebook scores three competing forecasts of the same 24 months across the whole metric family, then tests whether the differences are real.
A 24-month comparison table (2022 to 2023): the actual
demand plus three forecasts, model_a (accurate but missed one promotion spike), model_b
(systematically low), and naive (seasonal-naive benchmark). Ready to grade, no model fitting needed.
- ●Bias: ME is near zero for model_a but +496 for model_b and +827 for naive (both under-forecast).
- ●The flip: model_a wins on MAE (218 vs 496) but model_b edges it on RMSE (532 vs 564), the missed spike.
- ●Skill: MASE 0.26 and 0.60, both beat the naive benchmark (1.00).
- ●Significance: A vs B is not significant (p ≈ 0.90); A vs naive is (p ≈ 0.0001).
Forecast Evaluation in Machine Learning & AI
Modern forecasting keeps these metrics but changes how you compute them, over many windows, many series, and whole predictive distributions rather than single points.
| Idea | What it adds | Where it shows up |
|---|---|---|
| Rolling-origin backtesting | Re-forecast at many cutoffs (expanding or sliding window), not one split, time-series cross-validation | The standard way to evaluate any forecasting model |
| Probabilistic scoring | Grade the whole predictive distribution, not just the mean | Pinball / quantile loss, CRPS, interval coverage |
| Competition metrics | Blend and weight the classics across many series | M4 (OWA of sMAPE + MASE), M5 (WRMSSE) |
| Global-model evaluation | Aggregate error fairly across thousands of series of different scales | Weighted / scaled metrics for retail, energy |
| Guarding the test | Avoid leakage and look-ahead in the backtest pipeline | Same discipline as Chapters 105 and 126 |
Evaluation itself is a research topic. The move is toward probabilistic forecasting, judging models by proper scoring rules like CRPS that reward well-calibrated uncertainty, not just a point, and toward rolling-origin backtesting as the honest default. Forecasting competitions (M4, M5) have repeatedly shown that simple, well-evaluated statistical models rival elaborate ones, which is why the humble metrics in this chapter still decide who wins.
Grade the forecasts in Python
The companion notebook computes the residuals and ME (bias), the scale-dependent MAE/MSE/RMSE (and shows the MAE-vs-RMSE flip), the percentage MAPE/sMAPE, the naive-relative MASE and Theil's U, runs a Diebold-Mariano test between models, checks the winner's residuals with Ljung-Box, and lays out one verdict panel, all in a few lines of numpy and pandas.
View opens the rendered notebook instantly.
Open in Colab runs it live. To run locally, install numpy, pandas,
matplotlib, scipy, statsmodels, and openpyxl.
🎓 Key Takeaways
- ✓The metric is a loss function: choosing one is choosing which errors you refuse to tolerate.
- ✓ME is bias, not accuracy: a near-zero ME can still hide large errors that cancel.
- ✓MAE vs RMSE: RMSE (always at least MAE) punishes big misses hardest and can flip the ranking, as it did here.
- ✓Benchmark against naive: MASE and Theil's U below 1 is the bar every forecast must clear.
- ✓Test the gap: Diebold-Mariano tells you whether one model is really better or just lucky.
Practice Challenges
Five exercises on the forecast comparison. Full solutions are in the companion solutions notebook.
Bias vs accuracy
Compute ME, MAE, and RMSE for each model. Which one is biased, and how do you know?
Why RMSE beats up big errors
Show RMSE is always at least MAE and explain model_a's large gap.
Percentage traps
Compute MAPE and sMAPE. When does MAPE mislead, and what fixes it?
Beating the benchmark
Compute MASE and Theil's U. Which models beat the naive forecast?
Is the difference real?
Run Diebold-Mariano for A vs B and A vs naive. Interpret the two p-values.
Solutions notebook
All five challenges worked in code, bias versus accuracy, why RMSE punishes big errors, the percentage traps, beating the naive benchmark, and the Diebold-Mariano significance test, each with a short explanation.
Quiz: Test Yourself
Eight questions on the error metrics and comparing models. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
You can now decompose a series, forecast it, model its volatility, and grade the result. Time to put it all together. Case Study: Forecasting Retail Sales opens Forecasting Case Study with a full end-to-end forecasting project, from raw data to an evaluated, deployed forecast.