Observability at Scale: Reducing Alert Noise Without Missing Silent Quality Failures
An AI platform team supports 60 production models: 38 predictive, 15 generative, and 7 agentic workflows. Monitoring generates about 400 alerts per week; 70% are acknowledged as noise. Average time to acknowledge an alert is 4.5 hours, and mean time to resolve quality incidents is 2.3 days. Last month two incidents were missed: a fraud model's recall fell 18 points over nine days, and a support agent began returning empty citations for 6% of answers. Both had alerts that were muted during a release freeze. Each model has a different owner and dashboard, and there is no shared severity taxonomy. The team wants to reduce alert volume while still catching silent failures. Service-level objectives exist for uptime but not for model quality. The on-call rotation has six engineers and already pages on infrastructure alerts.
1. How would you redesign alerting and severity to cut noise while catching slow quality degradations?
Read the worked answer
Start with a service catalog: tier each model by customer impact, decision value, and reversibility. Tier 1 models such as fraud and credit get quality SLOs and paging; Tier 3 gets daily digests. Replace per-metric threshold alerts with composite health scores and leading indicators. For each model, define 3-5 metrics: input drift, prediction distribution, calibration or accuracy where labels exist, and an outcome proxy. Use anomaly detection with seasonality and require two consecutive breaches or a severity-weighted score before paging. Deduplicate alerts by model and root cause; route to owners. Target fewer than 40 actionable alerts per week and under 30 minutes to acknowledge Tier 1. Keep a separate slow-burn alert for gradual drift that would have caught the 18-point recall decline over nine days. Review muted alerts during release freezes; require a named risk acceptance, not a blanket mute.
2. What signals would have detected the two missed incidents earlier, and how would you instrument them?
Read the worked answer
The fraud recall drop is a delayed-label quality failure; instrument a labeled holdout evaluated weekly and alert on a 5-point drop over seven days, with a leading indicator from override or review outcomes. The empty-citation failure is a generative quality failure; instrument citation coverage, empty-response rate, retrieval recall on a sample, and groundedness on a daily sample. Add synthetic probes after each deploy and canary checks for generative endpoints. Use input and embedding drift to catch upstream data changes. Do not rely on uptime SLOs; quality SLOs are separate and must be owned. The instrumentation should log trace IDs so a bad answer can be tied to a retrieval or prompt version.
3. Define an incident response model for model quality incidents with six on-call engineers.
Read the worked answer
Define severity levels: SEV1 for customer harm or regulatory exposure, SEV2 for quality degradation, SEV3 for noise or single-user issues. Each model has a primary and secondary owner, a runbook, a rollback plan, and a communication channel. Tiering limits pages so six engineers can cover the rotation. On-call handles Tier 1 pages; Tier 2 goes to the model owner during business hours. Require post-incident review within five days with action items. Track mean time to detect, acknowledge, and resolve. The missed incidents show that release-freeze muting is a control gap; require review of muted alerts and a freeze exception process with a named approver. Quality incidents should feed back into the evaluation sets and alert thresholds.