Monitoring, Observability, and Drift Management
Domain 6 tests monitoring across service, data, model, and business layers, plus drift detection and retraining triggers.
What to monitor
- Service: latency, throughput, errors, saturation, queue depth.
- Data: schema, null rate, range violations, freshness, feature distribution.
- Model: prediction distribution, confidence, calibration, and delayed-label accuracy.
- Business: conversion, fraud loss, defect escape rate.
Drift rules
Data drift is a change in inputs; concept drift is a change in the input-to-label relationship; prediction drift is a shift in outputs. Use a fixed baseline window and statistical tests such as PSI, KS, or KL divergence, but set thresholds from observed variation, not arbitrary constants. Without labels, drift is a signal to investigate, not proof of degradation.
Worked example
A credit model shows PSI 0.25 on income and 0.05 on debt ratio. The label delay is 60 days. Reasoning: income drift is above the 0.2 alert threshold, so check whether an upstream pipeline changed units or a real economic shift occurred. Compare schema and source freshness first. If the shift is real, launch shadow scoring and schedule retraining with recent labels; validate subgroup fairness before promotion. Do not retrain immediately on unlabeled drift alone.
Traps
- Monitoring uptime only and calling it MLOps.
- Alerting on every metric, causing fatigue.
- No baseline or seasonal adjustment.
- Confusing data drift with concept drift.
- Retraining without a validation and fairness gate.