CCOA (IJCAHPO): Forty-Minute Pipelines and an 18 Percent Flake Rate

Where the time goes. With no dependency graph, every PR builds and tests all 900-odd packages, most of which the change cannot affect. There is no remote cache, so identical builds repeat across runners. Integration tests are serialized within a job, and the 11-runner count is fixed rather than derived from the work. Pulling and building images on every run adds fixed minutes. On top of that, 18% of runs are thrown away entirely, which is roughly 4-8 minutes of wasted compute per PR plus a human re-run cycle measured in hours.

Redesign:

• Affected-only builds using Nx, Turborepo, or Bazel with a dependency graph, so a change to one service runs its build plus its dependents' tests.
• Remote build and test cache keyed on content hashes; a cache hit should skip compilation, not just downloads.
• Split integration tests by shard across a dynamic matrix, sized to the measured work rather than a fixed 11.
• Ephemeral databases per shard via Testcontainers or a template database, so tests are isolated and parallel-safe.
• Container layer caching with a pinned base image and dependency layer before source copy.
• Contract tests at the service boundary instead of spinning up the full stack for every PR.

Target p50 PR feedback under 10 minutes, p95 under 15.

Flakes as defects. Instrument the test runner to record pass-on-retry and quarantine those tests immediately with an owning team and a 14-day fix SLA. Run quarantined tests 20 times nightly to measure true flake rate. Replace sleep-based waits with deterministic polling, pin clocks and random seeds, and record external HTTP fixtures. Set a flake budget of under 2% and publish it next to the build success rate. Deleting a flaky test is better than rerunning an unstable suite, because reruns hide real regressions.

Decouple deploy from release. Ship code behind feature flags so a deploy is reversible in seconds by flipping a flag, not by re-running a pipeline. This alone converts most rollbacks into a configuration change.

Progressive delivery with automated analysis. Canary 5% of traffic for 10 minutes, compare error rate, p99 latency, and saturation against the baseline, then promote to 25%, 50%, 100% with bake periods. Gate promotion on measured signals, not on a human watching a dashboard. Roll back automatically on a breached threshold; the on-call engineer confirms rather than initiates.

Fix the migration pattern, not the rollback. Adopt expand-contract for every schema change: release one adds a nullable column and dual-writes; release two backfills and switches reads; release three removes the old column. Each step is independently deployable and rollback-safe because no release requires the previous one to be undone. A destructive migration in the same release as application code is what makes the current rollback fail; prohibit it with a CI check that flags DROP, ALTER TYPE, and NOT NULL additions without defaults.

Rollback as a tested path. Rollback of a canary is a weight shift. Rollback of a full deploy is a redeploy of the previous image, which should take under 5 minutes if images are prebuilt and the pipeline is warm. Rehearse rollback in staging monthly and record its duration as a tracked metric.

Raise deploy frequency. Three deploys a week is a symptom of cost per deploy being too high. Smaller batches with canaries reduce change failure rate and make each deploy cheaper, which is the opposite of the intuition that fewer deploys are safer.

Measure the four DORA metrics from pipeline and incident data, not surveys.

• Lead time for change: commit timestamp to production timestamp, median and p90.
• Deployment frequency: count of production deploys per week, by service.
• Change failure rate: deploys requiring rollback, hotfix, or causing a Sev-2 or above, divided by total deploys.
• Time to restore: from incident open to service restored, or from failing canary to rollback complete.

Add two supporting measures so the headline numbers cannot be gamed: pipeline duration at p50/p95, and flake rate with a quarantine count. Also track percentage of deploys that required a manual approval step, which tends to predict the other four.

Set a baseline before changing anything, then review monthly. Expect change failure rate to fall before deployment frequency rises; frequency follows confidence.

On the freeze. A blanket five-day freeze before month-end concentrates risk rather than removing it. All deploys queue up behind the freeze and land together afterwards, which is exactly the batch size that produces incidents during the financial close. Replace it with risk-tiered controls: low-risk changes (flags off by default, static assets, non-financial services) deploy continuously through month-end; medium-risk changes require a canary and an on-call owner present; high-risk changes (schema migrations on billing, pricing logic, anything touching the ledger) are blocked during the close window and scheduled with a named rollback owner. That keeps the freeze on the two or three changes that can actually break month-end while letting the other 90% of the monorepo move. Publish the tier definitions in the repo so the decision is not relitigated per deploy.