Weekly Retraining With Regulated Approval Gates
A payments company retrains six fraud and risk models every week. The pipeline reads 11 sources: three BigQuery tables, four Pub/Sub streams landed to Cloud Storage, two vendor CSV drops, and two on-prem databases replicated through Datastream. A typical run processes 2.4 TB and must finish within a 5-hour window. Last month a schema change in a vendor feed silently replaced 17% of merchant category codes with nulls; the model still passed its AUC gate and was deployed. Auditors require reproducible training data snapshots, a named approver for every production deployment, and one-click rollback to the previous model. The team uses Vertex AI Pipelines, Cloud Build, and Vertex AI Model Registry, but approvals are currently handled in chat. The next audit is in 30 days.
1. Design a reproducible weekly pipeline with data validation and lineage. Where should the pipeline block?
Read the worked answer
Use Vertex AI Pipelines with pinned container images in Artifact Registry and explicit parameters for data snapshot, code commit, and training configuration. Ingest each source into a versioned landing zone, then write immutable BigQuery snapshots or Cloud Storage snapshots for the run. Datastream and Pub/Sub landings should carry ingestion timestamps and source offsets.
Insert validation before training: schema contracts, null-rate thresholds, category-domain checks, row-count deltas, and distribution tests. The 17% null merchant-category event should fail a hard gate, not merely warn. Block the pipeline before model training if validation fails; do not let a model train on bad data and rely on an AUC gate downstream.
Record lineage in ML Metadata: source snapshot IDs, validation results, feature definitions, training image digest, hyperparameters, and output model artifact. Store the model in Vertex AI Model Registry with the data snapshot and code commit as metadata. This gives auditors a reproducible chain from source to deployed model.
2. How would you implement a named-approver gate and one-click rollback without relying on chat?
Read the worked answer
Split the pipeline into train/evaluate and deploy stages. After evaluation, pause the pipeline or create a deployment request in a controlled system such as Cloud Build with a manual approval trigger. The approver must authenticate with their own IAM identity, and separation of duties should prevent the model developer from also approving deployment. Record the approver, timestamp, model version, evaluation report, and data snapshot in an immutable audit table.
Use Model Registry aliases such as candidate, production, and previous. Deployment changes the endpoint traffic split or alias only after approval. One-click rollback means reassigning production to previous and shifting traffic back through an audited Cloud Build job, not editing an endpoint by hand.
Test rollback in a game day and measure recovery time. The auditors should be able to see who approved what, which artifact was deployed, and which artifact replaced it, without reading chat logs.
3. Why did the AUC gate fail to catch the vendor schema change, and what gates would you add?
Read the worked answer
AUC is an aggregate ranking metric. If 17% of merchant category codes become null, the model may still rank most transactions well, especially if other features carry signal, so aggregate AUC can stay flat or even improve slightly. AUC also says nothing about schema validity, feature coverage, or business costs.
Add hard data-quality gates before training: schema compatibility, null-rate limits per feature, category-domain checks, and source row-count deltas. Add slice-based model gates after training: performance by merchant category, transaction size, and region, plus calibration and false-positive cost at the operating threshold. Require a feature-importance or drift check for the affected feature.
Finally, use a shadow or canary deployment with business metrics before full production. A model that passes AUC but degrades a critical merchant segment should not deploy. The approval record should show that these gates ran and passed, not just that one aggregate metric did.