The Drift Alert Nobody Owned
A logistics company runs a demand-forecasting model that drives warehouse staffing. It was trained on 2021-2022 data and has run unchanged for 18 months. A monitoring job computes a drift score weekly and emails a distribution list. For the past 11 weeks the score has exceeded the alert threshold. No one has acted; the list includes three people who have left the company. Forecast error (MAPE) has risen from 8% to 19%. Staffing decisions are made every Monday from the model output. Last month two warehouses were understaffed on peak days and one overstaffed for a week. The platform team says the model is 'stable' because it has not crashed, and the ML team says retraining is blocked because the feature pipeline owner left and no one has credentials to the upstream data source.
1. Identify the operational failures in this situation and rank them by urgency.
Read the worked answer
The failures are organizational as much as technical, and the ranking should reflect that.
- Highest urgency: no accountable owner for model performance. The alert goes to a dead distribution list, and no role is responsible for acting on it. This is the root cause; the other failures persist because no one owns them.
- High urgency: an alerting design that cannot succeed. An email to a list with no acknowledgment, escalation, or runbook is not monitoring. Eleven consecutive weekly breaches with no action means the alert is effectively disabled.
- High urgency: loss of the ability to retrain. The feature pipeline owner departed and credentials were not transferred. This is a bus-factor and access-management failure, and it converts a routine retraining into a blocked project.
- Medium urgency: the 'stable because it has not crashed' framing. Stability of the service is not stability of the model. MAPE moving from 8% to 19% is a doubling of error, and the staffing consequences are already visible.
- Medium: no linkage between model error and business outcome. Under- and overstaffing were observed but not traced back to the forecast.
Ranking by urgency: ownership first, because it unblocks everything else; then alerting and access; then the metric framing.
2. Design a monitoring and ownership structure that would have caught this earlier, and explain how each element addresses a specific failure above.
Read the worked answer
Elements, each mapped to a failure:
- Named model owner with a documented backup. Addresses failure 1. The owner is accountable for performance review, not just uptime, and the backup prevents a single departure from creating a gap.
- Alert routing with acknowledgment and escalation. Addresses failure 2. A drift breach opens a ticket assigned to the owner, with an escalation path if unacknowledged within a defined window. Repeated breaches without resolution escalate automatically.
- Performance monitoring on business-relevant metrics, not just drift score. Addresses failure 4. Track MAPE against a threshold (for example, act at 12%) and tie forecast error to staffing variance so the business consequence is visible.
- Access and knowledge continuity. Addresses failure 3. Credentials held in a managed secrets store with at least two authorized holders, and pipeline documentation reviewed on a schedule.
- A scheduled retraining cadence with a trigger. Addresses the 18-month staleness. Retrain quarterly, or sooner if MAPE crosses threshold, so retraining is routine rather than a crisis.
- A monthly review that connects model metrics to operational outcomes. Addresses failure 5.
The design principle is that every alert has an owner, an action, and a deadline. Monitoring that no one is required to act on is decoration.