Retraining Trigger and Data Drift Diagnosis
A demand-forecasting model is retrained monthly on 3 years of history. Recently, forecast error (MAPE) rose from 11% to 19% over six weeks. The team monitors input feature distributions with SageMaker Model Monitor and sees a shift in the promotion_flag and price features, but no missing values and no schema violations. A separate weekly batch job computes residuals. The business ran an unannounced pricing change three weeks ago. The monthly retrain pipeline pulls the latest 3 years, which now includes the post-change weeks, and validation MAPE on the most recent 4 weeks is 17%.
1. Distinguish the likely causes of the error increase and explain why the monthly retrain did not fix it.
Read the worked answer
The distribution shift in promotion_flag and price, coinciding with an unannounced pricing change, points to concept drift or a genuine regime change rather than a data-quality fault: there are no missing values or schema violations, so the pipeline is healthy but the relationship between price and demand has moved. The residual job showing rising error confirms the model, not the data, is now wrong.
The retrain did not fix it for two reasons. First, the training set is dominated by three years of pre-change data, so the model still fits the old price-demand relationship; three weeks of new regime is a tiny fraction of the window and gets washed out. Second, validation on the most recent 4 weeks gives 17% MAPE, which is better than the 19% observed but still far above the 11% baseline - the retrain improved fit to the new weeks but the evaluation window itself is short and noisy, so it is not clear the model has learned the new regime rather than overfit a few weeks.
The judgement: this is not a 'retrain more often' problem alone. It is a signal that the model's assumptions about price elasticity changed, and the fix requires either recency weighting, a feature capturing the pricing regime, or a segmented model.
2. Propose a concrete diagnosis and remediation plan, including how to change the retraining trigger.
Read the worked answer
Diagnose first: segment the error by product and price band to see whether the degradation is concentrated where prices moved. Compare residuals before and after the pricing change date to confirm the break. Check whether promotion_flag shifted because of the pricing change or independently, since two simultaneous shifts can mask each other.
Remediation options, in order of cost: (1) add recency weighting or shorten the training window so recent weeks carry more influence; (2) add a feature that encodes the pricing regime (e.g., price relative to a rolling baseline, or days since price change) so the model can condition on it; (3) if elasticity genuinely differs by segment, train per-segment models. Validate each on a time-based holdout that includes the post-change weeks, and report MAPE by segment, not just overall.
Retraining trigger: monthly cadence is too slow for a regime change. Add drift-triggered retraining - fire when MAPE over a rolling window exceeds a threshold (e.g., 14%) or when Model Monitor flags a sustained shift in a high-importance feature - and keep the monthly cadence as a floor. Also add an alert on the residual job so the team hears about a break within days, not at the next monthly run, and require a human review before promoting a model trained on a short post-change window, since it may overfit the new regime.