A 47-minute pipeline that keeps shipping bad code
Helios Health runs a 60-service monorepo on GitHub, built by Cloud Build triggers and deployed to dev, staging, and prod GKE clusters. Every build runs the full unit and integration suite and takes 47 minutes; developers merge about 35 PRs a weekday and the median PR waits 40 minutes for a verdict. Eighteen percent of test jobs fail and then pass on a retry, and the team's reflex is to press Retry. Nine of the last twelve production incidents came from changes that had passed CI. Separately, the ML team retrains a readmission-risk model weekly with a six-hour Vertex AI pipeline that writes the resulting model straight to a production endpoint. Compliance requires a signed, reproducible artifact and evidence that the artifact tested is the artifact deployed.
1. Cut time-to-verdict without reducing what the gate actually catches.
Read the worked answer
Diagnose before scaling: if the 47 minutes is 12 minutes of setup, 5 minutes of build, and 30 minutes of tests, a bigger machine buys almost nothing.
Split by what the gate must catch. On a PR: affected-target selection from the build graph (change impact analysis), remote build caching keyed on a content hash, and shard the test suite across 8 workers - 30 minutes of independent tests becomes roughly 4 minutes plus serial tail, so a realistic PR verdict is 10–12 minutes. Post-merge on main: the full suite, contract tests, and the slow integration environments. Nightly: the long-running soak and mutation suites.
The flakiness is the bigger defect. Eighteen percent flake with a Retry reflex means the gate's signal is noise, and it explains 9 of 12 incidents far better than coverage does. Quarantine flaky tests into a non-blocking suite within one business day of a second unexplained failure, publish flake rate per service as a tracked metric, and treat 'retry until green' as an incident, not a workflow. A fast pipeline that everyone learns to ignore is worse than the 47-minute one.
Cloud Build: use a private pool sized for your real concurrency, and cache container layers in Artifact Registry rather than rebuilding them per step.
2. Compliance wants proof that the artifact tested is the artifact deployed. Design that, and explain how it would have prevented the migration incident.
Read the worked answer
Build once, promote by digest. The pipeline produces one image, records its digest, and generates SLSA provenance and a vulnerability scan attestation. Staging and prod deploys reference the digest, never a tag, and never rebuild. Binary Authorization on the prod cluster enforces attestors that map to environments: only prod-deploy attestor signatures are admitted in prod, and only the prod promotion job holds that signing identity. This gives the auditor a chain - source commit, build ID, digest, attestation, admission - rather than a screenshot.
For the connection-pool outage, the root cause is usually coupling: the app and the schema moved together, so there was no artifact you could have deployed independently. Enforce expand/contract migrations as pipeline policy: release 1 ships the additive schema change (new column, nullable, dual-write) with the old code still valid; release 2 ships the code that reads it, canaried; release 3 removes the old column weeks later. Each step is its own digest, reversibly deployable, and the canary has a real rollback target.
Add a schema-change label that forces a load test at projected peak concurrency (8x) and a check that the connection pool size divided by max replicas stays under the Postgres limit.
3. The weekly retrain writes directly to the production endpoint. What changes?
Read the worked answer
Separate training from deployment - the six-hour pipeline currently has production write access, which means an evaluation bug is an incident.
The pipeline becomes: data validation, training, evaluation, registration, deployment, each with its own gate. Training writes to the Vertex AI Model Registry with metadata; it does not touch an endpoint. The evaluation step gates on overall and per-slice metrics (age band, comorbidity count) against the incumbent model, not against a fixed threshold - a model that is 0.5% worse overall but 15% worse on one slice should not ship.
Deployment is a traffic split on the production endpoint: shadow or 5% canary first, with the online monitoring metrics compared between variants, promoted by an automated job or a human depending on the risk tier. Keep the previous model version deployed for one-click rollback.
Guard training-serving skew explicitly: the feature transforms in the training pipeline must be the same code or the same Feature Store view used at serving time, and the pipeline should verify the training feature distributions against the last 7 days of served features. The single biggest failure mode in a weekly retrain is silent data drift, not model quality.