Notebook-and-cron retraining across three teams
Three data science teams share a Vertex AI project. Retraining today is a Cloud Composer DAG that shells out to Dataproc, then a data scientist runs a notebook by hand to train, then someone copies the model artifact to a Cloud Storage path that the serving code reads. Two incidents last quarter: a model was promoted that had been trained on a stale feature table, and a retrain could not be reproduced after the fact. Leadership wants weekly automated retraining with a human approval before production, plus line-of-business audit evidence of what data produced what model. One team also needs to pull a flat file from an on-prem SFTP server each morning.
1. Should the retraining move to Vertex AI Pipelines, stay in Composer, or both? Justify the split.
Read the worked answer
Both, with a clear division of responsibility.
Vertex AI Pipelines for the ML path. The incidents described are ML-lineage failures: stale feature table and irreproducible training run. Pipelines addresses both natively - each step declares typed inputs and outputs, artifacts are tracked, and every run records which dataset version fed which model. The training step becomes a containerized component, not a human in a notebook.
Composer for the external dependency and scheduling. The on-prem SFTP pull is exactly what Composer is good at: it has the operators, retry semantics, and connectivity for non-GCP sources, and it already exists. Have Composer fetch and land the file, then trigger the Vertex AI Pipeline for the ML steps.
If the team's only scheduling need were the ML pipeline, Cloud Scheduler plus a native pipeline schedule would be simpler than Composer. The SFTP requirement is what tips it. Do not rebuild the SFTP pull as a custom pipeline component just for uniformity - you would be reimplementing Composer badly.
2. How do you make retrains reproducible and stop the stale-feature-table incident recurring?
Read the worked answer
Reproducibility is a design property, not a log message. Four concrete moves:
- Pin the data. The feature-preparation step writes a versioned snapshot (date-partitioned table or a versioned feature view) and emits its identifier as a pipeline artifact. Downstream steps take that artifact as input, so they cannot silently read 'current' data.
- Pin the code. Container images referenced by digest, not by mutable tag. A
latesttag is the same class of bug as an unpinned feature table. - Pin the environment. Same image for training and for the eventual serving container where possible; if not, record both digests.
- Make the artifact the handoff. The serving path reads from the Model Registry, never from a Cloud Storage path a human copied into. That single change removes the copy step where the wrong artifact gets promoted.
For audit evidence, the pipeline run ID is the answer to 'what produced this model': it resolves to snapshot version, image digests, parameters, and evaluation metrics. Test this by having someone who was not on the project reconstruct last month's model from the run ID alone.
3. Design the approval gate and the trigger for weekly retraining.
Read the worked answer
Split the pipeline at the promotion boundary. The automated pipeline trains, evaluates, and registers the model with a non-production alias (candidate). A separate promotion step moves the production alias after approval. Serving reads the alias, so promotion is an alias flip, not a redeploy of unknown provenance.
The gate itself: Vertex AI Pipelines has no native human-approval node, so use one of two patterns - a pipeline step that waits on a flag (simple, but a long-parked run ties up tracking) or, better, keep promotion outside the pipeline as a small Cloud Build / Cloud Function job that a reviewer triggers from a ticket or a UI action after reading the evaluation metrics. The second keeps humans off the critical path of training and gives you an approval audit trail.
Triggering: a native pipeline schedule for the weekly cadence; for the SFTP-dependent team, Composer starts the pipeline after the file lands. Add a guard step that aborts if the training snapshot is older than expected or evaluation metric regresses beyond a threshold - an automated retrain that promotes unconditionally is just a faster way to ship a bad model.