Nightly Forecasts, a Manual Promotion Table, and a $600k Markdown
A regional grocer runs a legacy rules-based replenishment forecast for 1,200 stores and roughly 38,000 store-SKU combinations. Weekly forecasts drive automatic orders to three distribution centers. The legacy system's weighted MAPE is 34%, and stores complain about both stockouts on promoted items and dead stock. An in-house gradient-boosted model reaches 27% MAPE in offline backtests, and the supply chain VP wants it live before next quarter. Features come from a feature store; the promotion calendar is a table merchandising maintains by hand and refreshes "most weeks." The nightly scoring job runs at 01:00 and writes forecasts into the ordering system, which executes at 05:00 with no human review. Last spring a mis-keyed promotion week produced forecasts 2.3x normal for eleven SKUs across 400 stores; $600k of overstock was eventually marked down. A separate incident left 14 remodeled stores flagged as closed in the store master for nine days.
1. Design the guardrails you would require before this model is allowed to write orders. Be concrete about checks, thresholds, and ownership.
Read the worked answer
Treat the forecast as safety-critical, because it commits capital without a human in the loop. Guardrails in three layers:
1. Input contracts. A freshness SLA on the promotion table - block the run if the max update timestamp is older than 24 hours. Row counts and null rates within tolerance of the trailing four weeks. Referential integrity to the store master, including a check that "closed" stores are not silently excluded from ordering. Detection of new or unmapped SKUs. The stale promotion feed and the remodeled stores both fail here, which is the point.
2. Prediction sanity. Compare each batch against a naive seasonal baseline and against the prior run. Flag SKU-stores where the forecast moves more than a set percentage or exceeds a physical bound - shelf capacity, DC inventory, case-pack multiples.
3. Outcome monitoring. With the natural lag: forecast bias and error, stockouts, and overstock dollars by store and category, reviewed weekly.
Every check needs a named owner and a defined action. A monitor that alerts to nobody and triggers nothing is decoration, and it will be discovered as decoration after the next incident.
2. The VP wants all 1,200 stores switched at once to capture the benefit immediately. Lay out the rollout you would counter-propose, including how the system behaves when something fails.
Read the worked answer
Switching everything at once means the fallback is also the outage.
Shadow mode first. Run the new model in parallel with the legacy rules for four to six weeks, logging both. Compare at the SKU level, focusing on where they disagree and who turns out to be right, with specific attention to promotion weeks and the remodeled stores.
Staged cohorts. Roll out to matched store pairs - about 10% of stores - and expand on pre-agreed criteria. Keep a randomized holdout on the legacy system so the benefit is measured rather than asserted; a 7-point MAPE improvement in backtest often shrinks in production.
Fallback. It must be automatic and rehearsed. If an input check fails, the pipeline emits the legacy forecast and marks the batch "degraded" rather than shipping a number nobody trusts. Keep the last good forecast as a secondary fallback for late-breaking failures. Run the fallback in a game day before go-live, because an untested circuit breaker is a hypothesis.
Abort criteria, defined in advance. For example, bias beyond ±8% for two consecutive weeks triggers rollback. Writing this down beforehand prevents rollback from becoming a negotiation during a crisis, when everyone's incentive is to wait one more week.
3. You have inherited the aftermath of the $600k incident. What do you change in the operating model, not just in the code?
Read the worked answer
The coding fixes are the easy half. The incident was organizational.
Ownership. The promotion table is a hand-maintained artifact owned by merchandising but consumed as a data contract by forecasting. Write a formal SLA: named owner, update deadline (say, 48 hours before promotion start), and a downstream impact statement so the merchandising team understands that a late edit costs six figures. People comply with obligations they can see the consequences of.
Make failure legible. Hold a blameless review within a week, producing named changes: the freshness gate, a monthly reconciliation between the store master and the forecast universe, and credential/token expiry moved onto a calendar-owned rotation.
Debt with a business case. Automated promotion capture from POS and markdown systems, replacing manual entry. Fund it with the $600k.
Escalate the near-miss upward with the cost estimate. The overstock figure is the argument that pays for the guardrails; without it, the fixes compete with features and lose. Governance is what makes a deployment stay correct after the launch team moves on.