AI Operations, Monitoring, and Continuous Improvement
AI Operations, Monitoring, and Continuous Improvement
This domain tests running AI in production: observability, drift detection, incident response, retraining, cost control, and continuous improvement. Expect questions on monitoring metrics, alert thresholds, and change control.
Core rules and concepts
- Monitor four layers: infrastructure (latency, errors, availability), data (freshness, schema, drift), model (performance, calibration, subgroup metrics), and business (conversion, cost, override rate).
- Drift types: data drift is a change in inputs; concept drift is a change in the input-output relationship. Use statistical tests such as PSI or KS, but pair them with performance checks.
- Ground truth may arrive late. Use proxy metrics and sampled labels; define a maximum acceptable delay.
- Incident management: severity levels, on-call ownership, rollback, communication, and post-incident review. Model changes need versioning and change control.
- Retraining policy: triggers, validation gates, and approval. Continuous improvement includes feedback loops, cost optimization, and decommissioning.
Worked example
A fraud model's precision falls from 0.80 to 0.68 after a holiday campaign. Input PSI for transaction amount rises above 0.2; fraud patterns also shift. The team checks sampled confirmed labels, finds concept drift, and retrains on recent data. Before promotion, it validates precision and recall by segment and runs a canary release. Reasoning: drift alerts are signals, not automatic retrain commands; validation and change control prevent a worse model from reaching production.
Traps
Monitoring only uptime; ignoring delayed ground truth; auto-retraining without validation; no rollback plan; and failing to monitor cost and business outcomes.