Developing ML models
This domain tests model development on Vertex AI and Gemini: choosing the right starting point, training and tuning correctly, and evaluating honestly.
Core rules
- Baseline in ascending cost. Prompt-only Gemini, then few-shot prompting, then BigQuery ML or AutoML, then custom training, then fine-tuning. Each step must be justified by a measured gap.
- Vertex AI Training runs prebuilt or custom containers; distributed training uses multi-worker replicas, and the reduction server optimizes all-reduce for large models. Hyperparameter tuning is managed by Vizier.
- Tuning has distinct jobs. Supervised fine-tuning teaches format, tone and label behaviour from examples; distillation moves capability into a cheaper model; RAG and grounding supply fresh facts. Fine-tuning is not a reliable way to inject new knowledge.
- Evaluate on untouched data. Pick the metric that matches the cost of errors, such as precision, recall or F1 for imbalanced classes rather than accuracy, and compare candidates on the same held-out set.
- Explainability. Sampled Shapley, integrated gradients and XRAI on Vertex AI help confirm the model relies on sensible features.
Worked example
A support team has 800 labeled tickets across 12 categories and wants a cheap classifier. Reasoning: start with a Gemini few-shot prompt and score it on a held-out 20%. If the shortfall is formatting or label consistency, run supervised fine-tuning on the remaining 80% and compare on the same held-out set. If cost or latency still misses the target, distill into a smaller model. Training a bespoke model from scratch would consume the whole labeling budget without a proven gain.
Traps
- Fine-tuning before trying prompt engineering and few-shot examples.
- Reusing the test set across many tuning rounds, producing optimistic scores.
- Reporting accuracy on a heavily imbalanced dataset.
- Selecting a TPU or GPU whose framework support does not match the training code.
- Expecting fine-tuning to add product knowledge that changed last week.
- Training without versioning the dataset and the resulting model artifact, making results unreproducible.