Serving and Scaling Models
Two serving modes
Batch prediction writes to BigQuery or Cloud Storage and is right for large throughput, no latency SLO. Online prediction deploys to a Vertex AI Endpoint and answers per-request; it is right when a human or service waits on the answer. Questions that mention "nightly" or "millions of records" are batch; "sub-second" or "user-facing" are online.
Endpoint mechanics
- Import the model into the Model Registry, then deploy to an endpoint with a machine type, container image, and min/max replicas plus an autoscaling target (CPU utilization or GPU duty cycle).
- Dedicated endpoints support traffic splitting by percentage (must total 100), which is how you do canary and A/B rollouts. Shared/public endpoints do not - they serve one model with no split capability.
- Private endpoints with Private Service Connect keep traffic off the public internet; VPC-SC and CMEK are common add-ons.
- Custom prediction routines / custom containers when you need pre/post-processing at serve time.
- Explainable AI attaches feature attributions to online or batch predictions.
Traps
- Deploying a model only to discover the container lacks a health-check route or the expected request schema.
- Confusing endpoint-level autoscaling with pipeline-level parallelism.
- Setting min replicas to zero to save money while the SLO demands low p95 latency - cold starts break it.
- Trying to do a canary on a shared endpoint.
- Forgetting that deploying a new model version creates a new deployed model ID unless traffic is shifted.
Rule of thumb
Size min replicas for the baseline traffic you must serve within SLO, and let autoscaling absorb bursts. Then canary the new version at 10/90 traffic and watch error rate and latency before flipping to 100.
Worked example
A model must serve 200 QPS at p95 under 200 ms with periodic spikes. Deploy to a dedicated endpoint with min replicas covering ~200 QPS, max replicas sized for the spike, CPU target 60%, and a Private Service Connect attachment.