Scaling Prototypes into Production-Grade Models
From notebook to trained artifact
The exam pushes you past "it works on my sample." Know the training options: AutoML (no code, managed splits and tuning), BigQuery ML (train where the data lives, good for tabular baselines), custom training on Vertex AI with a prebuilt container or your own container image, and Vertex AI Pipelines when the steps repeat. Hyperparameter tuning uses Vizier - you set the search space, metric, and trial budget; it searches values, not architecture.
Concepts to have ready
- Distributed training: mirrored strategy for single-node multi-GPU; multi-worker with Reduction Server to cut all-reduce bandwidth on large GPU clusters; TPUs for very large batch workloads.
- Data hygiene: temporal splits for time series, hashing for stable splits, checking leakage between train and eval, handling class imbalance with the right metric (precision/recall, AUC-PR) rather than accuracy.
- Training-serving skew: the same transforms must run at prediction time - bake them into the model (preprocessing inside the container) or into a shared component.
- Experiment tracking: Vertex AI Experiments for runs and metrics; save datasets as versioned artifacts.
- Capacity: GPU/TPU quota is per region and per project; request before the training deadline.
Traps
- "Improve accuracy" answered with a bigger model when the real issue is a leaky split or a bad label.
- Assuming hyperparameter tuning will fix a poor feature set.
- Sending a notebook to production as-is instead of containerizing the training step.
- Forgetting that a failed long job still consumes quota and cost.
Rule of thumb
Always establish a cheap baseline first - BigQuery ML or AutoML on a sample. Move to custom training only when you can name the metric gap and the reason the managed option cannot close it.
Worked example
A tabular churn model in a notebook hits 0.71 AUC. Before writing distributed training code, run BigQuery ML logistic regression and boosted trees on the full table; if the gap is small, productionize the BQML model and spend the engineering time on features, not infrastructure.