Automate and orchestrate ML pipelines
Choosing the orchestrator
- Vertex AI Pipelines (Kubeflow Pipelines or TFX SDK) - the ML-native choice. Components are containerised steps; inputs/outputs are tracked as artifacts with automatic lineage; runs support caching, parameterisation, scheduling and retries. Templates are stored in Artifact Registry.
- Cloud Composer (Airflow) - the enterprise choice when the DAG spans systems outside Vertex AI (on-prem, SAP, long human approvals, many non-ML tasks).
- Workflows - lightweight API/service orchestration; no ML artifact tracking.
- Cloud Scheduler - cron triggers; Eventarc / Pub/Sub - event triggers (new object in GCS, Model Monitoring alert, BigQuery row insert).
- Cloud Build + Cloud Deploy / Artifact Registry - CI/CD: build the training image, compile the pipeline template, submit the run.
Traps
- Caching surprises. Pipeline caching keys on component spec and inputs; identical components are skipped. A question about a step that "did not re-run after a code change" usually means the image tag was not bumped.
- A notebook is not a component. It must be packaged as a container or a
PythonPackageJob-style component before it can join a DAG. - Retraining loops. Model Monitoring alerts should publish to Pub/Sub or Cloud Logging, with a Cloud Function/Workflow that triggers the pipeline - not a hand-run notebook.
- Scheduling is delegated. Vertex AI Pipelines schedules are backed by Cloud Scheduler; the service account needs the right roles or runs fail silently.
- Composer vs Pipelines is the most common either/or question; answer with the scope of the DAG.
Rule of thumb
If everything in the DAG is an ML step producing artifacts → Vertex AI Pipelines. If the DAG must also call enterprise systems or hold for approvals → Composer, calling Vertex AI Pipelines as one task.
Worked example: Nightly retrain → evaluate → deploy only if the metric beats the current production alias. Build it as a pipeline whose final component conditionally registers a new model version and updates the production alias; trigger with Cloud Scheduler.