I85PREP
⌘ K
GCP ML Engineer study materials
GCP ML Engineer study resources

GCP ML Engineer (Google Cloud Certifications): Serve and scale models

GCP ML Engineer · Work through your exam, one topic at a time.

All guide sections

Serve and scale models

The serving decision tree

Batch prediction runs a job against a Cloud Storage file or BigQuery table and writes results to storage or BigQuery. It is cheap and has no endpoint, no autoscaling and no SLA. Online prediction deploys a model to a Vertex AI Endpoint for low-latency requests.

Endpoint mechanics you must know:

  • A deployed model has a machine type, optional accelerators, and min/max replicas with an autoscaling target (CPU utilisation, GPU duty cycle, request count).
  • Traffic splitting routes a percentage of requests to different deployed model IDs - the mechanism for canary and blue/green rollouts.
  • Endpoint types: shared public, dedicated public (static IP, lower blast radius), and private (Private Service Connect / VPC peering) for private-IP-only requirements.
  • Explainability (Sampled Shapley, integrated gradients, XRAI) can be enabled per deployed model.
  • Custom containers must implement health, predict and (for GPU frameworks) the right routes; serving frameworks such as TorchServe, TF Serving or vLLM usually wrap your model.

Traps

  • "No latency requirement, 10 TB of data" → batch prediction, not an endpoint. Many candidates over-deploy.
  • Min replicas above zero removes cold starts but bills continuously; max replicas protects the backend but moves failure to 429s.
  • Traffic split is per request, not per user - sticky behaviour needs routing outside the endpoint.
  • Deploying one model to several endpoints multiplies cost; share an endpoint via traffic split or deploy once.
  • Batch prediction does not use endpoints or autoscaling and cannot do per-request explanations the same way.

Rule of thumb

Ask two questions: is there a real-time consumer? and is volume large and predictable? Real-time + spiky → endpoint with low min and high max. Non-real-time + bulk → batch.

Worked example: A nightly churn score for 40 million customers → batch prediction reading a BigQuery table and writing scores back to a BigQuery table. A fraud check inside a payment flow → endpoint with min replicas 2 to avoid cold starts.