I85PREP
⌘ K
GCP ML Engineer study materials
GCP ML Engineer study resources

GCP ML Engineer (Google Cloud Certifications): Scale prototypes into ML models

GCP ML Engineer · Work through your exam, one topic at a time.

All guide sections

Scale prototypes into ML models

From notebook to production-grade training

The prototype moment is when data no longer fits in memory, training takes hours, or results must be reproducible. The levers:

  • Custom training on Vertex AI - prebuilt containers (sklearn/XGBoost/PyTorch/TF) or your own container; the training package goes to Cloud Storage or Artifact Registry; the job writes artifacts to a baseOutputDir.
  • Distributed training - MirroredStrategy for one machine, MultiWorkerMirroredStrategy across workers. For bandwidth-bound AllReduce, enable the reduction server, which aggregates gradients through a tree rather than a ring.
  • Accelerators - GPUs, or TPUs for very large matrix workloads; Persistent Resources keep an interactive dev VM warm.
  • Hyperparameter tuning - Vertex AI Vizier: metric + goal, search algorithm (Bayesian default), max trial count, parallel trial count, early stopping.
  • AutoML vs custom - AutoML for tabular/image/text when speed and baseline quality matter; custom when you need control of architecture, loss, or preprocessing.

Traps

  • Hyperparameter tuning runs training trials - it is billed per trial, and parallel trials multiply cost while reducing statistical quality per trial.
  • Reduction server addresses bandwidth, not latency. A question about slow step time on many small GPUs is a different fix.
  • Training-serving skew from preprocessing. Feature engineering done in the notebook must be packaged as a component or custom container, otherwise serving receives raw features.
  • Accelerators do not fix input starvation. If the GPU sits idle, fix the tf.data/BigQuery reader first.
  • Data splits and leakage - scaling or imputing before splitting leaks test information.

Rule of thumb

Profile before scaling: if GPU utilisation is low, fix the input pipeline; if it is high but step time is dominated by communication, add reduction server or fewer, larger workers.

Worked example: A 16-worker GPU job scales to 8 workers with almost no slowdown and GPU utilisation under 40%. Diagnosis: AllReduce is bandwidth-bound. Enable the reduction server and keep the worker count rather than adding more GPUs.