Monitor AI solutions
Two different monitoring problems
Operational health - endpoint request count, latency, error rate, CPU/GPU utilisation in Cloud Monitoring; prediction errors and stack traces in Cloud Logging; admin activity in Cloud Audit Logs. This tells you the service is up.
Model quality decay - Vertex AI Model Monitoring attached to a deployed model. It compares live traffic against baselines using three objective types:
- Training-serving skew - live feature distribution vs the training baseline.
- Prediction drift - live distribution vs a previous window of live traffic.
- Feature attribution drift - change in how much each feature contributes to the prediction (explainable models).
Monitoring samples requests, writes statistics to a destination (BigQuery), and alerts via email, Cloud Logging or Pub/Sub. A baseline dataset with the training schema is required.
Traps
- Skew vs drift. Skew = training vs serving. Drift = today vs yesterday. Questions describe one and name the other.
- Drift needs no labels; quality does. True accuracy degradation can only be measured once ground truth arrives, which is delayed - so drift is the proxy.
- Monitoring needs a deployed endpoint and the right sampling rate; excessive sampling costs money and can itself affect latency.
- Generative AI evaluation is separate - continuous evaluation, Rapid Evaluation and AutoSxS in Vertex AI Studio/Gen AI eval service, not classic tabular Model Monitoring.
- Fairness/bias is assessed with model evaluation slices and Fairness Indicators, not with drift metrics.
Rule of thumb
Map the complaint to the metric first: slow or 5xx → Cloud Monitoring; inputs look different → skew/drift; answers are worse → ground-truth evaluation with a labelled sample.
Worked example: A model trained on January data is deployed in June; request volume is stable but inputs contain a new device category. That is training-serving skew (live vs training baseline), and prediction drift will also fire as June traffic diverges from the earlier live window. Automate a retraining trigger from the Pub/Sub alert.