Detecting Silent Drift in a Demand Forecast
A logistics company forecasts daily package volume per region to staff sortation centers. The model is a gradient-boosted regressor retrained monthly on two years of history. For six weeks, forecast error has crept up in three coastal regions while overall error looks normal. Operations noticed because they were understaffed on Mondays. Root cause is suspected to be a shift in e-commerce behavior plus a new regional carrier that changed delivery patterns. The team has no per-region monitoring, only a global MAE dashboard, and labels (actual volumes) arrive with a one-day lag. Retraining takes eight hours and requires manual approval.
1. Why did the global dashboard fail to surface this, and what monitoring should replace it?
Read the worked answer
Why it failed. A global MAE averages over all regions. Three regions degrading while the remaining regions stay stable - or improve - can leave the aggregate flat or even slightly better. Aggregation masks localized drift, and the model's errors are correlated within a region, so a regional bias is invisible in a pooled mean. The Monday understaffing is the operational consequence: the error is concentrated on a specific day-and-region slice that the dashboard never examines.
Replacement monitoring.
Segmented accuracy: compute MAE and bias (mean signed error) per region, and per region-by-day-of-week, since staffing decisions are made at that granularity. Alert on each segment independently.
Bias monitoring specifically: systematic over- or under-forecasting is more damaging for staffing than random error, and a signed-error metric catches it earlier than MAE.
Input drift: monitor distributions of the top features per region - order volume trends, carrier mix, day-of-week patterns - using a baseline from training. Input drift appears before label-based error does, which matters given the one-day label lag.
Prediction drift: track the distribution of forecast outputs per region; a sudden level shift warrants investigation even before actuals arrive.
Use SageMaker Model Monitor with a baseline captured at training time, configured per segment, and route alerts to the operations team that owns staffing, not only to the ML team.
2. Given the one-day label lag and an eight-hour retrain with manual approval, propose a response process that limits operational damage.
Read the worked answer
Short-term containment. Because actuals arrive one day late, the team can detect a regional bias within roughly 24 hours of it starting. Add a fast correction layer ahead of retraining: a per-region bias adjustment computed from the trailing error, applied to forecasts until the model is retrained. This is cheap, reversible, and buys time - but it must be bounded and logged, because it can mask a genuine model failure.
Escalation. Define tiers: a bias alert in one region triggers analyst review; sustained bias across multiple regions or a threshold breach triggers an emergency retrain. Pre-authorize the emergency path so manual approval does not add hours to an eight-hour job.
Retraining. Shorten the loop where possible - incremental or more frequent retraining on recent data, and a faster training configuration validated in advance so the eight-hour figure is not the emergency path. Include the new carrier as an explicit feature if it explains the shift; otherwise the model cannot adapt regardless of retrain frequency.
Post-incident. Add the Monday-by-region slice to the standard monitoring set, and record the detection-to-response timeline to test whether the process actually meets the operational window staffing requires.