A 47-Minute Pipeline Nobody Trusts

Start by quantifying the size of the problem: 24% failure × 70% rerun-pass ≈ 17% of all runs are false red. That is the number to move.

1. Shared Postgres across eight parallel runners (largest). It produces exactly the signature observed - timing-dependent failures that pass in isolation, spread across unrelated suites. Measurement: parse JUnit XML for the trailing 30 days into (commit, test, job, runner) rows. Compute the flip rate per test: failures followed by a pass on the same commit ÷ total executions, with a minimum sample of 50 runs. Then correlate failure rate with concurrency: replay the same set of commits with the fan-out forced to 1 and compare. If the failure rate collapses, the database is the cause; if not, the ranking is wrong and you have spent a day, not a quarter.

2. Test pollution and order dependence. Measurement: run the suite with randomized order (--sequence.shuffle) ten times on a known-good commit and count tests whose result changes. Failures that cluster when suite A precedes suite B point at leaked rows or global state.

3. Sleep-based synchronisation. sleep(30) is a fixed bet on how loaded the runner is. Measurement: per-test duration p50/p99 ratio; tests whose p99 exceeds 3× median are load-sensitive. Confirm by grepping for sleeps and fixed timeouts and by checking whether failures correlate with job queue depth.

4. Staging end-to-end suites. 22 of 47 minutes, and they test a shared environment with external dependencies. Measurement: compute their flip rate separately from unit tests; they usually dominate the tail even when they are numerically fewer.

5. Cache misconfiguration. This causes no false failures but wastes roughly 3 minutes × 9 jobs per run, and it lengthens the feedback loop that makes people override. Measurement: cache hit rate, which is currently effectively 0% because github.sha changes on every commit.

Definition. Flake rate per test = runs that failed and then passed on retry of the same commit ÷ total runs, computed over a trailing 14-day window with at least 50 executions. Below 50 executions the number is noise, so leave the test alone.

Trigger. Any test above 2% flake rate, or any test that failed for three different authors in a week, is quarantined automatically by a scheduled job that edits the test manifest.

Mechanics. Quarantined tests move to a separate @quarantine job that is non-blocking but still runs and still reports. The job opens a ticket and assigns it to the code owner of the touched package (not "the team"). SLA: 10 working days to fix, or delete the test with a linked reason. Deletion must be explicitly permitted - a test nobody will fix is worse than no test, because it trains people to ignore red.

Anti-graveyard controls.

• Quarantine count is itself a tracked budget: more than 15 tests or more than 1% of the suite stops feature work for a hardening sprint.
• Age is reported, not just count; anything over SLA escalates to the engineering manager weekly.
• No silent retries in the gating job: a retry may run for diagnosis, but the first outcome is still recorded as a failure data point, otherwise retries hide the metric.
• Review the quarantine list in the weekly ops meeting with a named owner per item.

Dashboard metric. Pipeline false-red rate = runs that failed and passed on rerun ÷ total runs, target < 2%, deadline six weeks - a 17% × six-week burn-down is realistic with the database isolation and cache fixes, and a 2% target is below the level at which people stop trusting the signal.

Order the work by minutes saved per engineering hour invested, not by how interesting it is.

Week 1 - cheap, high-certainty wins.

• Fix the cache key to hashFiles('**/package-lock.json') plus the Node version. Saves about 3 minutes per job across nine jobs, removes no signal at all, and is a one-line change.
• Replace sleep(30) with a poll-until-ready helper with a 60-second cap. This changes flake behaviour and shortens the common path.
• Split the single workflow into a fast lane (lint + unit, target under 10 minutes) that runs on every push, and a slow lane (end-to-end) that runs on the merge queue and nightly. Path filters plus affected-package selection keep the fast lane honest on a monorepo.
• Introduce a merge queue. It eliminates the N-developers-rerunning-the-same-main pattern and catches semantic conflicts that per-PR CI cannot see.

Week 2 - the expensive fix that actually matters. Give each runner its own schema (CREATE SCHEMA test_$CI_NODE_INDEX) or an ephemeral container per job. This is the change that addresses the top-ranked cause, so schedule it deliberately rather than squeezing it in.

Trade-offs accepted. Path filters and affected-package selection can miss cross-package breakage; mitigate by running the full suite on the merge queue and nightly, and by making the package graph explicit. A merge queue adds queue latency at peak; that is a fair trade for removing override culture. Moving end-to-end out of the per-push path means defects in integration surface are found later - acceptable only if the nightly is genuinely triaged, so assign a rotating owner for nightly failures before you make the switch.