Monitoring, optimizing, and maintaining ML solutions
This domain covers what happens after launch: detecting drift and quality loss, tuning cost, and keeping models and endpoints current. The exam expects monitoring to be layered and connected to retraining, not just dashboards.
Core rules
- Monitor three layers. Infrastructure metrics (latency, errors, CPU/GPU, replica count) in Cloud Monitoring; data and prediction drift in Vertex AI Model Monitoring, which compares serving traffic to a training baseline for skew and drift; and quality once labels or user feedback arrive.
- Set baselines and thresholds deliberately. Monitoring without a baseline or alert threshold produces noise; align thresholds with the business cost of a bad prediction.
- Log safely. Request and response logging aids investigation but must exclude or mask PII and respect retention rules.
- Close the loop. Drift alerts should trigger the retraining pipeline, with an evaluation gate and canary before promotion.
- Optimize cost continuously: batch instead of online where possible, right-size replicas, prune unused endpoints and models, consider distillation or quantization, and use committed use discounts for steady load.
- Track model and API versions. Gemini models and Vertex AI features are versioned and deprecated; pin versions and plan migrations.
Worked example
Model Monitoring reports feature drift in average order value, and the fraud classifier's precision has fallen. Reasoning: drift can come from an upstream change, so first check the Dataflow job and the distribution of raw inputs. If the input is legitimate and behaviour has genuinely shifted, retrain on recent labeled data, validate on a fresh time-based holdout, then canary the new version. Update the monitoring baseline to the new training data only after promotion.
Traps
- Watching only infrastructure metrics and never data or prediction drift.
- Launching monitoring without a baseline, then drowning in alerts.
- Expecting accuracy monitoring without labels; use proxy metrics or delayed labeling.
- Leaving monitoring configuration stale after a feature schema change.
- Assuming Vertex AI retrains models automatically; it does not.
- Missing model deprecation notices and discovering them in production.