Automating and Orchestrating ML Pipelines
What to orchestrate
A production ML pipeline chains: data validation, feature/transform, training, evaluation with a quality gate, model registration, deployment, and monitoring setup. On Google Cloud, Vertex AI Pipelines (Kubeflow Pipelines v2 or TFX) is the ML-native orchestrator: components are containerized steps, data moves as artifacts with URIs, and Vertex ML Metadata records lineage. Cloud Composer (Airflow) is for orchestrating across many systems, not a single ML DAG - questions that hinge on "already standardized on Airflow" point to Composer, ones about "ML lineage and caching" point to Vertex AI Pipelines.
Mechanisms worth memorizing
- Caching keyed on component inputs, so re-running a pipeline skips unchanged steps.
- Conditionals and ParallelFor for evaluation gates and per-shard work.
- Retries and timeouts per step for transient failures.
- Templates and parameters compiled once, run with different inputs and schedules.
- Triggers: native pipeline schedules, Cloud Scheduler for cron, Eventarc or Pub/Sub for event-driven runs (new file landed, new BigQuery partition).
- Build chain: source in Git → Cloud Build → image in Artifact Registry → pipeline step.
Traps
- Passing DataFrames between steps instead of GCS/BigQuery URIs - artifacts must be durable.
- Putting orchestration logic inside a notebook rather than in components.
- Using Container Registry instead of Artifact Registry for new builds.
- Expecting Vertex AI Pipelines to retrain automatically when drift happens - something must trigger it.
- Confusing a HyperparameterTuningJob (one step) with a pipeline (many steps).
Rule of thumb
Every step should be idempotent, containerized, and communicate only via artifacts. If a step can't be re-run safely from its inputs, it doesn't belong in the pipeline.
Worked example
A nightly retrain needs to skip training when the new data has no meaningful change. Build a pipeline whose first component emits a data_changed boolean artifact, gate training with a conditional, gate deployment on an evaluation threshold, and trigger the run from a Cloud Scheduler cron.