Monitoring, observability, and maintenance
Monitoring, observability, and maintenance tests whether you can detect degradation and operate models after deployment. Observability has three layers: infrastructure (CPU, memory, latency, errors), data (schema, drift, freshness), and model quality (accuracy, fairness, business KPI).
Core rules
- Send endpoint logs and metrics to Azure Monitor and Application Insights; use Log Analytics for queries and alerts.
- Define baselines from training or a stable production window. Monitor data drift and prediction drift on a schedule; collect ground truth when available.
- Alert on symptoms that require action: p95 latency, 5xx rate, missing features, drift beyond threshold, and quality below SLA.
- Automate retraining only with a validated trigger and a promotion gate; keep the previous model deployable.
Worked example
A demand model shows PSI 0.25 on a key feature and p95 latency 450 ms. The team checks whether the drift is a data pipeline bug or real market change. They find a missing holiday feature, fix the pipeline, backfill, and retrain. The new model must beat the current model on a holdout and pass a shadow deployment before traffic shifts. Latency is fixed by scaling the endpoint.
Traps
- Monitoring only CPU and memory.
- Alerting on every metric without thresholds or ownership.
- Retraining automatically without a quality gate.
- No baseline, so drift is undefined.
- Ignoring feature freshness and upstream data-contract failures.