Forecast Drift With a Thirty-Day Label Lag
A utility company operates a day-ahead demand forecasting model for 40 substations, run once nightly against three years of history. At launch the fleet-level MAPE on the evaluation harness was 11.2%; over the last six weeks it has drifted to 19.4%. Monitoring consists of service uptime, request latency, and a nightly job that checks that predictions are non-null. There is no accuracy dashboard, because ground truth (metered load) arrives with a 30-day settlement lag and about 3% of meters are subsequently re-read and revised. A new electric-vehicle charging tariff launched in two of the 40 substations eight weeks ago. The model is retrained quarterly on a fixed three-year window with unchanged features, and nobody has examined feature distributions since launch. Forecasters have started overriding the model manually for the two tariff substations, which the team only learned about last week.
1. Design a monitoring scheme that would detect this degradation despite the 30-day settlement lag.
Read the worked answer
With labels a month stale, monitoring has to run on three layers, only one of which uses ground truth.
Layer 1 - operational. Uptime, latency, NaN or null predictions, prediction volume per substation, and the override rate. The existing checks live here and caught nothing, which is expected: they cannot see accuracy. Override rate is worth adding precisely because it surfaced the problem informally - forecasters noticed before the monitoring did.
Layer 2 - drift and data quality, available immediately. Per substation and per horizon: PSI or a KS test on each input (temperature, calendar, tariff flag), null rates, out-of-range checks. Watch the prediction distribution itself too - a forecast that flattens or shifts level is often the first visible symptom. Set thresholds from a rolling 12-week baseline and alert per segment, not fleet-wide: a fleet MAPE can look tolerable while 2 of 40 substations are badly wrong and being manually corrected.
Layer 3 - delayed accuracy, revision-aware. Store every prediction immutably with its issue timestamp and re-score when settlement data lands, then re-score again after the 3% meter revisions so the figure is stable and comparable across vintages. Report MAPE by substation, horizon and week, alongside a seasonal-naive baseline so genuine model degradation is separated from genuinely harder weeks.
The governing rule: a fleet-level alert on a 30-day-old number is both too slow and too coarse. Segment-level drift plus a naive-baseline comparison yields a usable signal within days.
2. Root-cause the drift and decide between retraining and a model change.
Read the worked answer
Split the degradation before choosing a remedy.
The EV tariff in 2 of 40 substations eight weeks ago is the obvious suspect, and the timing fits: drift started six weeks ago, consistent with a couple of weeks of post-launch data accumulating. Compute MAPE per substation for the last eight weeks. If the increase is concentrated in those two, this is a structural break in a small segment. Retraining on three years of pre-tariff history will not fix it - the model has never seen that regime, and a quarterly retrain folds a few weeks of the new pattern in against three years of the old, so the new behaviour is swamped rather than learned. The fix is representational: tariff-aware features (charging-load indicators, day-type interactions), possibly a separate model or an explicit segment adjustment for those substations, and a serious review of whether one fleet model is appropriate at all.
If instead the degradation is broad across all 40, look at the inputs first. A drifted weather feed or a changed holiday calendar is more likely than a genuine shift in demand behaviour, and neither is fixed by retraining on the same broken feed.
Do both checks on the same post-launch window, and hold out the eight post-launch weeks so any fix is measured on the regime it has to handle, not on three comfortable years.
3. Propose a retraining and promotion policy that fits a system with structural breaks, manual overrides and a one-month label lag.
Read the worked answer
Three changes: cadence, gates and shadowing.
Cadence. Quarterly is too slow for a system with structural breaks. Move to monthly retraining on a rolling window, and treat retraining as trigger-driven as well: alert if segment MAPE exceeds the naive baseline by a margin for two consecutive weeks, or if drift thresholds fire. Because labels lag 30 days, a retrain started today can only be validated on data settled a month ago - accept that limitation and pair it with the drift layer rather than pretending accuracy monitoring alone will catch it.
Gates. Promote only if the challenger beats the champion on a rolling-origin backtest over 6–12 months, with per-substation and per-horizon floors, not just a fleet average. Include the naive baseline so a bad month does not make a mediocre model look good. Fail the gate outright if input drift or null-rate checks fail, regardless of accuracy.
Shadow. Run the challenger in shadow for at least one full label cycle before it takes traffic, and score it against the champion on the same settled window. Keep the champion warm for rollback.
Overrides. Log manual overrides as first-class signals with reason codes. Forecasters overriding two substations for weeks is a monitoring channel the team did not design and should formalise - it is free ground truth about where the model is failing.