AI Operations, Monitoring, and Maintenance
What this domain tests
What you watch after launch, how you tell benign input change from genuine degradation, and how you decide when to retrain rather than roll back.
Core rules
- Monitor four layers: infrastructure (latency, error rate, GPU saturation), data (schema, ranges, null rates, distribution shift via PSI or KS), model (accuracy when labels arrive, calibration, PR-AUC), and business (conversion, cost, manual override rate).
- Drift types: covariate or data drift (input distribution moves), concept drift (the input-to-output relationship changes), label drift, and plain upstream pipeline breakage.
- Labels usually arrive late, so pair delayed ground truth with leading proxy indicators such as score distribution shift or override rate.
- Retraining triggers: scheduled cadence, performance threshold breach, drift threshold, or incident review.
- Version data and model together; use shadow, canary, blue-green, or A/B rollouts and keep the previous artefact ready.
- Page humans on user-impacting SLO breaches; file tickets for slow drift warnings, or alerts get ignored.
Worked example
A key feature's population stability index rises from 0.05 to 0.32 while measured accuracy is unchanged. Reasoning: values above roughly 0.25 conventionally indicate a major shift, so investigate the upstream source first - a units change, a new vendor feed, or a broken join. Retraining immediately would bake the corrupted distribution into the model.
Traps that cost marks
Monitoring uptime only. No pipeline to capture ground truth. Retraining on drifted data without root-cause analysis. Setting drift thresholds with no baseline. Alerting on every metric until the on-call engineer mutes the channel.