Retraining Cadence vs. Drift You Cannot See Yet

Two distinct regime changes, and they need different responses. Weekly retraining addresses neither well on its own.

Regime 1 - level shift (competitor exit). Baseline demand stepped up for ~30% of SKUs. A model trained on 24 months sees mostly pre-shift history; with monthly retraining it has had 4 updates, but 20 of 24 months still reflect the old level. The model is regressing toward a stale mean. The SKU breakdown confirms this is where the error lives.

Regime 2 - new spike generator (promotion engine). Promotions create a demand pattern the model has never seen. Two months of history is thin, and if promotions are not a feature, the model treats spikes as noise or as level shifts, which corrupts the baseline estimate for those SKUs.

Will weekly retraining on the same features fix it? Mostly no.

• It helps regime 1 only slowly: more frequent updates still average over 24 months, so the stale level persists. The fix for a level shift is recency weighting or a shorter/segmented training window, not frequency.
• It cannot help regime 2 at all: without a promotion feature, the model has no way to attribute spikes. Worse, weekly retraining on spike-contaminated data can make the baseline less stable, as the model chases promotional noise.
• Weekly retraining increases operational load and the chance of overfitting to a noisy recent window.

Diagnosis: the degradation is structural (regime change plus a missing feature), not a staleness problem that cadence alone solves.

For the level shift:

• Shorten or weight the window. Train on 6-9 months with exponential recency weighting, or fit a level-shift correction per SKU. A 24-month window is a liability when the level moved 4 months ago.
• Segment the model. The 30% of affected SKUs behave differently; a separate model or a SKU-level regime indicator lets each group use appropriate history. At minimum, add features that let the model detect the shift: competitor presence, market share proxies, or a change-point indicator.
• Validate on post-shift data only. Backtesting on pre-shift months will look great and mislead. Evaluate on the last 8 weeks.

For promotions:

• Add promotion features: promotion flag, discount depth, duration, channel, and a forward-looking promotion calendar (known in advance - this is legitimate, not leakage, because the calendar is available at forecast time).
• Model the spike explicitly rather than letting it leak into baseline: predict baseline and promotional uplift separately, or use a feature that isolates promotional days.
• Handle the thin history: 2 months of promotions is not enough to learn uplift reliably for all 5,000 SKUs. Pool information across SKUs by promotion type, or use a hierarchical/global model with SKU-level adjustments.
• Clean the training labels: promotional days in history distort baseline estimates; either model them or exclude them from baseline fitting with explicit handling.

Cadence: keep retraining frequent (weekly is reasonable for the level component) but pair it with recency weighting and a change-detection trigger that forces a retrain when SKU-level residuals shift persistently. Cadence is a safety net, not the mechanism.

The 1-day lag is short - use it. The failure here was 10 months of silent drift, which means monitoring was absent or too coarse.

Leading indicators (same day, no labels needed):

• Forecast vs. actual residual tracking per SKU, with a control chart. A persistent positive bias on 30% of SKUs would have flagged within days of the competitor exit. Alert on sustained bias (e.g., 5 consecutive days same-sign), not single-day error.
• Prediction distribution drift: monitor the distribution of forecasts and inputs per SKU. A level shift shows up as a shifted forecast distribution even before errors accumulate.
• Feature availability and range checks: catch a missing promotion calendar or a new promotion type the model has never seen.

Label-based metrics (1-day lag):

• MAPE and bias by SKU segment, not just overall. The overall 11% → 19% masked that errors were concentrated; segment-level tracking would have localized it immediately.
• Stratified error by promotion vs. non-promotion days to catch regime 2.

Process:

• Backtest on rolling recent windows at each retrain, and compare against the incumbent before promotion. A challenger that cannot beat the incumbent on the last 8 weeks should not ship.
• Change-point detection on residuals to trigger investigation and retraining automatically.
• Named owner and alert thresholds tied to business impact (e.g., alert when segment MAPE exceeds 15% for 3 days), with a documented response.
• Post-incident review after this event, recorded with the model version, so the next regime change is recognized faster.