Retraining Triggers for a Demand Model With Week-Delayed Labels

Combine three signals; none is trustworthy alone.

1. Label-free input drift (feature distribution shift) - cheap, fast, high false-positive rate, which is exactly why it is ignored today. Use it as a screen, never a trigger.
2. Delayed performance measured once labels land at day 7 - the only signal that reflects reality, but it lags.
3. A shadow evaluation: the candidate model scores the last N days offline and is compared to the live model on the same rows once labels arrive.

Policy: drift alert opens a candidate retrain; the candidate deploys only if shadow performance beats the live model by a pre-agreed margin (e.g., 2% WAPE improvement) on the most recent labelled window. Otherwise the drift alert is recorded and closed with a reason. That converts 20 ignored alerts a month into a small number of decisions with evidence.

Also separate seasonal retrains (calendar-driven, at known assortment changes) from drift-driven retrains; mixing them is why the monthly schedule felt necessary. State the false-negative risk explicitly: a 7-day label lag means a genuinely broken model runs at least a week before the gate can see it, so keep an input-drift hard alert that pages a human even without labels.

Gate - three conditions, all checkable by pipeline, no email:

• Candidate beats or ties the live model on the held-out labelled window.
• Segment check: no top-20 SKU group degrades more than X% even if the aggregate improves (aggregate wins hide localized damage).
• Lineage complete: data snapshot ID + commit SHA + eval report attached to the registry version.

A named approver clicks in the pipeline UI; the approval is logged with the eval artifacts, which is what the email thread never captured.

Rollback must be an alias flip, not an image rebuild:

• Deploy behind a stable endpoint name; promote by moving the alias to an already-registered, already-built image.
• Pre-pull the previous image into the scoring cluster so the flip is a restart, not a pull.
• Rehearse the flip quarterly and record the actual time; a 15-minute promise you have never measured is a guess.

Expect the first version of the gate to be noisy - two or three consecutive runs where the candidate ties the live model. Fix that with a larger shadow window, not by lowering the bar.

The risk is concentrated in what the metric does not cover, not in the metric itself.

• Feedback loops: order suggestions change ordering behavior, which changes the sales data the model learns from. Auto-promotion means the model can drift into a self-reinforcing pattern with no human in the loop.
• Distribution shift inside the eval window: a candidate trained on a promo-heavy month can look excellent and collapse the following week.
• Silent scope change: an automated pipeline will happily retrain when the feature set or SKU assortment changed, producing a model that is 'better' on a different problem.

A defensible middle ground: auto-promote to a shadow/canary stage with full automation, and require human approval only for the step that moves traffic in production. Canary at a small traffic slice with automatic revert on metric regression gives most of the speed with a bounded blast radius. Keep human sign-off where the cost of a bad week is business-visible, and be explicit that this is a risk-appetite decision the business owner should make, not the ML team alone.