Serving and scaling ML models
This domain tests how you expose models and keep them responsive and affordable: online endpoints, batch prediction, traffic management and GenAI-specific serving controls.
Core rules
- Match mode to latency need. Vertex AI Endpoints for real-time requests; Batch Prediction for large asynchronous jobs; streaming pipelines when results must flow into another system.
- Autoscaling uses minimum and maximum replicas plus a utilization target. Minimum replicas remove cold starts at a cost; maximum replicas cap spend but can throttle bursts.
- Traffic split on one endpoint enables canary releases and A/B tests without building a second system.
- Isolate and secure. Private endpoints, VPC Service Controls and CMEK for sensitive workloads; separate endpoints or projects per trust boundary.
- GenAI serving has its own levers: provisioned throughput for predictable high volume, context caching to avoid resending long prompts, safety filters, and Vector Search for retrieval at request time.
- Respect request limits. Payload size and timeouts mean large inputs belong in Cloud Storage with a reference, or in a batch job.
Worked example
A new ranking model must reach production with low risk during a peak sales period. Reasoning: deploy the new version to the existing endpoint alongside the champion, route 5% of traffic to it, and compare error rate and p95 latency in Cloud Monitoring against the champion. Ramp to 100% only after the canary clears thresholds, then remove the old version. Autoscaling keeps a minimum of two replicas so latency does not spike during scale-out.
Traps
- Sending a one-off 10-million-row scoring job through an online endpoint.
- Assuming scale-to-zero is invisible to users; cold starts break latency SLOs.
- Discovering regional GPU quota only during launch.
- Deploying a model without pinning the version, so the artifact changes silently later.
- Setting maximum replicas too high and exhausting project quota.
- Placing endpoints in a region that violates data residency requirements.