Automating and orchestrating ML pipelines
This domain covers making training repeatable: pipelines, triggers, artifact lineage and gated deployment. The exam favours automated, versioned workflows over scripts run by hand.
Core rules
- Vertex AI Pipelines (Kubeflow Pipelines or TFX) is the ML-native orchestrator: each step is a container, artifacts flow between steps, and Vertex ML Metadata records lineage from dataset to model.
- Cloud Composer suits orchestration spanning many systems and non-ML dependencies. Cloud Workflows chains lightweight services, Cloud Scheduler handles time triggers, and Eventarc handles events such as a new object in Cloud Storage.
- Gate deployment. Use conditional components so a model is registered in Model Registry and deployed only when evaluation metrics clear a threshold.
- Cache and parameterize. Caching keyed on inputs avoids retraining unchanged steps; parameters keep one pipeline usable across environments.
- Make components idempotent and safe to retry, so a transient failure does not corrupt artifacts.
- Deliver pipeline code like application code, with Cloud Build and a source repository.
Worked example
Retraining must start whenever new labeled data lands in Cloud Storage. Reasoning: configure an Eventarc trigger on object finalization that calls a Cloud Run function to submit the Vertex AI Pipeline. The pipeline validates the data, trains, evaluates, and only when the metric beats the champion does a conditional step register the model and deploy it to a canary endpoint. Lineage makes the decision auditable, and caching keeps unchanged preprocessing from re-running.
Traps
- Orchestrating with cron plus a shell script, losing lineage, retries and gating.
- Auto-deploying every trained model without an evaluation gate.
- Embedding credentials in pipeline code instead of using a service account.
- Writing large datasets or models into pipeline parameters or metadata instead of Cloud Storage.
- Non-idempotent steps that duplicate rows or artifacts when retried.
- Skipping component caching, which multiplies cost and runtime on every run.