Demand Forecast Drift After a Promotion System Change
FreshCart operates 340 grocery stores and uses a daily SKU-level demand forecast to drive replenishment. The model was deployed 14 months ago and initially achieved 10.8% weighted MAPE. Over the last two months overall MAPE has risen to 18.9%, and in stores that adopted the new loyalty app it is 31%. Six weeks ago marketing replaced the weekly paper flyer with personalised app offers. In the feed, promotion_flag is now 1 for almost every SKU, and discount_depth has a much wider distribution than in training. The model still uses weather, holiday, price, store cluster and the old flyer fields. The API is healthy, with 80 ms latency and no errors. Store managers report both overstock in chilled categories and stockouts in household products. No retraining has occurred since launch. The replenishment lead asks whether this is a data problem, a model problem or a business-process problem, and what should be done this week versus this quarter.
1. Separate the symptoms into data drift, concept drift and pipeline or semantic change. What evidence would you collect?
Read the worked answer
The rise is too large and too segmented to be ordinary noise. The strongest signal is a semantic change: promotion_flag is now almost always 1 and discount_depth has shifted, so the model's inputs no longer mean what they meant in training. That is data drift, but it is also a pipeline or semantic break because the old flyer fields no longer represent the promotion mechanism. Concept drift may also be present if customers respond differently to personalised offers, but we cannot confirm it until we observe post-change sales and labels. I would collect distribution comparisons such as PSI or KS for each feature before and after the change; performance by store and category; delayed actuals after the six-week change; a check that promotion_flag and discount_depth are computed by the same logic as training; and store-level operational data on overstock and stockouts. The 31% MAPE in loyalty-app stores suggests the app rollout is a separate segment effect, not just a global drift.
2. Propose an immediate mitigation for this week and a longer-term monitoring and retraining plan for this quarter.
Read the worked answer
This week: do not wait for a full retrain. Fix or disable the misleading promotion features and fall back to a simpler baseline forecast for affected SKUs, especially chilled and household categories. Ask marketing for the exact definition and history of the new app offers, and backfill if possible. Add manual review for high-value or high-perishability SKUs. This is containment, not a cure. This quarter: rebuild the feature pipeline to represent personalised offers properly, including offer type, audience, exposure, discount depth and timing; retrain on data that includes the post-change period; and validate with time-based backtests. Run the new model in shadow mode against the current forecast. Add monitoring for feature distributions, segment-level MAPE, bias and stockout or overstock rates, with alerts and a named owner. Set a retraining cadence and a rollback path.
3. The business asks whether to switch off the model. What criteria would you use for rollback, and what is the fallback?
Read the worked answer
I would use explicit criteria, not a vague sense that the model is bad. Roll back or switch to the fallback if segment-level MAPE exceeds a pre-agreed threshold for two consecutive weeks, if the promotion feature cannot be trusted, if stockouts in critical categories rise above baseline, or if the model's forecasts are biased in one direction. The fallback should be a simple, transparent baseline, such as seasonal naive or a moving average with manual promotion overrides, not the broken model. A partial rollback by category is sensible: chilled and household may need different treatment. The decision should be made by the replenishment lead with the data team, documented, and reviewed weekly. Switching off the model entirely is not automatically safer; a naive forecast can also cause waste.