One Pipeline, Three Rebuilds, Six Hours to Production
Kestrel Payments runs a single pipeline for a 60-service platform. On every merge to main it compiles all services (28 minutes), runs unit tests (14 minutes), builds a Docker image tagged with the pipeline ID, and then, for each of dev, staging and production, re-runs the build and re-tags the image. Integration tests run only after the staging deploy, serially, taking 40 minutes, and the suite retries up to three times; 22% of runs need a retry. Production deploys require approval from any member of a 30-person engineering group, granted on average 2 hours after the request. Commit-to-production is 6.5 hours in the median, and a severity-1 hotfix takes about 5 hours to reach production. Twice last quarter, staging and production ran images built from the same commit but with different dependency resolutions; one of those caused a 40-minute outage.
1. Explain why rebuilding the artifact per environment is dangerous, and describe the promotion model you would replace it with, including how versioning and provenance should work.
Read the worked answer
The failure is not the rebuild, it is that the rebuild is not the same build
Re-running the build in dev, staging and production means three independent dependency resolutions, three base-image pulls and three chances for a floating version (latest, ~> ranges, unpinned transitives) to produce different bytes. The pipeline tests one artifact and ships another; the tested and shipped things share only a commit SHA. That is precisely the outage described - same commit, different resolution - and it means the test results were valid for an artifact that never reached production.
Promotion model
- Build once, on merge to main, producing an immutable artifact.
- Record its digest (image digest, not tag) and attach an SBOM plus provenance: which source revision, which builder, which lockfile, which base image digest.
- Every later stage refers to the digest. 'Deploy to staging' means
image@sha256:..., never:latestand never:build-4271. - Environments differ only in runtime configuration and secret material - never in artifact content.
- Pin everything the build touches: base images by digest, dependency lockfiles committed, plugin and toolchain versions. Treat reproducibility as a build-system property and verify it with a periodic rebuild-and-compare job, not with a deploy step.
- Gate promotion on the recorded results for that digest rather than re-running tests against a fresh image. A promoted artifact's evidence should travel with it.
Edge cases worth naming
If a configuration genuinely must be compiled in, build one artifact per configuration, promote each separately, and never rebuild silently inside a stage. Hotfixes follow the same path: build once from the release branch hotfix commit, promote the digest through the same gates, with an expedited approval path rather than a bypassed one.
2. Redesign the pipeline stages to cut commit-to-production time. State which stages parallelise, which must remain serial, and how you would retire the 30-person approval group without removing control.
Read the worked answer
Build stage - serial, one producer
Build only the services affected by the change, using the repository dependency graph. A 60-service monorepo should not compile 60 services for a one-line change; expect 28 minutes to drop to single digits in the common case. Emit the image digest, SBOM and provenance once.
Test stage - fan out, then fan in
- Unit tests sharded across runners by module.
- Integration tests run against an ephemeral environment created from the just-built digest, in parallel with unit tests - not after a staging deploy 40 minutes down the line. This removes the serial 40 minutes from the critical path entirely.
- Contract tests verified on both consumer and provider sides.
- Fan-in gate: promotion requires every shard green and the recorded evidence attached to the digest.
Promotion stage
dev → staging → production, each referring to the same digest. Health checks and canary analysis before full traffic genuinely must remain serial, and so must the promotion order itself. Nothing else needs to be.
Retiring the 30-person approval group
The 2-hour wait for any one of 30 people is queue time with no control value - the approver cannot meaningfully assess 60 services in five minutes. Replace it with policy-as-code: required automated checks, change-freeze windows, protected-path rules, separation-of-duties on schema changes, and a risk score that auto-approves low-risk changes while routing high-risk ones to a named on-call approver. The control is the enforced check, not the human signature.
Expected effect
6.5 hours compresses mainly because integration testing leaves the critical path and the approval queue disappears. The hotfix path needs its own pipeline that skips no gates but runs them concurrently.
3. Twenty-two percent of runs need a retry. What does that number actually tell you, and what retry and quarantine policy do you set?
Read the worked answer
What 22% means
It does not mean the pipeline is 78% reliable. It means at least one test is nondeterministic on roughly one run in five, and the true rate is higher because retried failures become invisible. Three different things are hiding behind the same number and they need different responses:
- Test-level flake - timing, port collisions, shared fixtures, order dependence.
- Environment instability - DNS, registry pull limits, resource contention on runners.
- Real intermittent product defects - races, timeouts, retry logic that double-charges. This is the dangerous category and it looks identical to the first two.
Policy
- Tag every retry with a reason code at the moment of retry. Without that data categories 1 and 3 are indistinguishable.
- Track failure rate per test, not per run. A test failing 5% of the time over 200 runs is a quarantine candidate regardless of how green the pipeline looks.
- Quarantine rather than retry silently: move the test to a non-blocking quarantine lane that still runs and reports, assign an owner and a 14-day expiry, and fail the build when the expiry passes. Quarantine count and age are the health metrics.
- Fail on new flake. A test that flakes and is not on the known list breaks the build, so flake is paid for at the moment it is introduced rather than amortised forever.
- Retry only infrastructure-classified errors, capped at one attempt, and count them visibly.
- Never retry the deploy or the promotion decision - only test execution.
The success measure is not a green pipeline. It is the escape rate of intermittent defects plus the age of the oldest quarantined test.