Monitoring AI Solutions
Three signals to separate
- Training-serving skew: distribution of live feature values versus the training baseline. Detects broken upstream data and feature pipelines.
- Prediction drift: live feature distribution versus a previous window of live traffic. Detects the world changing without a pipeline bug.
- Model quality / outcome drift: accuracy or business KPI decay. Slowest and most important, and usually requires delayed ground truth.
Vertex AI Model Monitoring attaches to an endpoint or a batch job, needs a baseline (training data or a saved distribution), sampling rate, thresholds, and a Cloud Monitoring notification channel. Explainable AI adds feature attribution, which lets you see which feature drifted or drove a prediction.
Operational layer
Cloud Monitoring gives endpoint metrics (request count, latency, error rate) and alerting policies; Cloud Logging plus request/response logging on the endpoint captures payloads for debugging and for computing quality later; Model Registry keeps evaluation metrics per version so a regression is visible at promote time.
Traps
- Monitoring turned on with no baseline - nothing to compare against.
- Enabling model monitoring but never enabling prediction logging, then being unable to investigate.
- Confusing "monitoring alerts fired" with "model retrained" - alerting and retraining are separate; the alert must trigger the pipeline.
- Watching only infrastructure metrics and declaring a degraded model healthy.
- Treating drift as automatically bad: input drift may be harmless if quality holds.
Rule of thumb
Alert early on inputs (drift and skew are fast signals), alert later on outputs (quality is slow), and only retrain after the drift is confirmed against validation data.
Worked example
A fraud model's precision drops while latency is fine. Check request/response logs and feature attribution: if a key feature's live distribution shifted, the upstream job broke - fix the pipeline before retraining. If inputs look normal and precision still fell, the relationship changed; retrain on recent data and canary the new version.