The Forecast Model Nobody Can Explain
A regional retailer deploys a demand-forecasting model to set weekly stock levels for 900 SKUs. The model improves aggregate forecast error by 15% versus the prior spreadsheet method. But store managers report that for a subset of ~60 seasonal SKUs the model consistently over-orders, filling back rooms with unsold goods, while it under-orders fast-moving staples. The data team says the model is 'working as designed.' Investigation reveals: (1) the training data came from 3 years of order history, which reflects past manager overrides, not true demand; (2) the seasonal SKUs were introduced 18 months ago, so they have thin history; (3) promotions are recorded in a separate system that was never joined to the model's input table. Finance is asking whether to retire the model or keep it.
1. Diagnose why the model fails on seasonal SKUs and staples specifically, using the three data facts. Distinguish problems caused by the data from problems caused by the modeling approach.
Read the worked answer
Seasonal SKUs - over-ordering. Two data problems compound. First, thin history: 18 months of data for a product with an annual cycle gives you roughly one and a half seasonal cycles, which is not enough to learn a seasonal pattern; the model likely falls back on a generic average and over-orders when the true seasonal peak is narrower than average. Second, promotions not joined: seasonal SKUs are heavily promoted, and because promo signals are missing from the input table, the model can't distinguish 'demand spike from a promo' from 'baseline demand,' so it bakes promo-driven volume into baseline and keeps ordering at that level after the promo ends. That is a data failure (missing feature / join gap), not a modeling failure.
Staples - under-ordering. This is a label problem. The target variable is order history, which reflects manager overrides. If managers historically trimmed orders on staples (e.g., to control shrink or cash), the model learned the trimmed number as 'demand' and now under-orders relative to true sell-through. The model is faithfully reproducing a biased label.
Data vs. modeling: the seasonal over-order is primarily a data problem (missing promo feature, insufficient history for the seasonal SKUs). The staples under-order is also a data problem (target leakage of human behavior into the label). The modeling approach is not obviously wrong - it is being fed a target that isn't demand and inputs that omit a key driver. Retiring the model would discard a genuinely better forecaster for the 840 SKUs where it works.
2. Finance asks 'retire or keep.' Recommend a path and specify the data changes that must happen before the model is trusted again.
Read the worked answer
Recommendation: keep, but scope it and fix the data contract before trusting it. Retiring a model that beats the spreadsheet on 840 SKUs throws away real value; the failure is concentrated and diagnosable.
Immediate (this week):
- Re-scope. Apply the model only to SKUs with sufficient history and no unmodeled promo dependency (the ~840), and revert the ~60 seasonal SKUs to a manager-reviewed method until fixed. This stops the back-room bleed now.
Data fixes required before re-expansion:
- Redefine the target. Replace 'order history' with a demand signal that strips out manager overrides - e.g., sell-through/point-of-sale where available, or reconstruct true demand from stockouts + sales. Document this as the label definition.
- Join promotions. Bring the promo system into the input table as an explicit feature (promo flag, depth, duration). This is the single highest-leverage fix for the seasonal SKUs.
- Handle thin history explicitly. For new/low-history SKUs, use a hierarchical or pooled approach (borrow seasonal shape from similar SKUs) rather than letting the model guess, and flag low-confidence SKUs for human review.
- Add a monitoring layer. Track error by SKU segment, not just in aggregate - the aggregate 15% hid a segment-level regression.
Governance: require a named owner for the input table and a data-quality check (promo join coverage, history depth) before any SKU enters the model. 'Working as designed' is the problem: the design was never validated against true demand.