The Flaky Gate That Everyone Learned to Rerun
A team of nine engineers runs a CI pipeline on every pull request. The test stage takes 22 minutes and has failed on 31 of the last 100 runs. Of those 31 failures, engineers judged 24 to be unrelated to the change and simply clicked rerun; 19 of those reruns passed on the second attempt. The remaining 7 failures were real defects. The team lead proposes raising the retry count to 3 automatic retries before reporting failure, arguing this will 'cut noise'. Meanwhile, a senior engineer notes that two of the seven real defects were found only on the second or third attempt of a flaky test, and one production incident last quarter traced back to a defect that had been dismissed as flaky. PR merge time has risen from 4 hours to 2 days.
1. Evaluate the lead's proposal to add three automatic retries. What does the data suggest about the likely outcome, and what would you recommend instead?
Read the worked answer
The retry proposal treats a symptom, not the cause. With 24 of 31 failures being non-defect noise, retries would convert most of them into green builds, but they would also mask the 2 real defects that only surfaced on later attempts, and they leave the 22-minute feedback loop and the 2-day merge time untouched. Worse, automatic retries encode the habit of ignoring red into the pipeline itself.
The data actually says the suite has a ~24% false-failure rate, which is far too high to be treated as normal. Recommendation:
- Quarantine, don't retry silently. Identify the specific flaky tests (the data implies a small number of repeat offenders), move them to a non-blocking suite, and file tickets with owners. Keep retries at zero for the blocking suite.
- Make flakiness visible. Publish a flaky-test leaderboard and fail the pipeline if quarantine grows.
- Attack the 22 minutes. Parallelize or split the suite; faster feedback shrinks the window in which developers context-switch and merge time falls.
- Distinguish signals. A test that fails intermittently on the same commit is a test defect; a test that fails deterministically is a product defect. The pipeline should tell them apart, not blur them.
If a temporary retry is used as a bridge, cap it at one, log every retry as a metric, and set an expiry date tied to fixing the named tests.
2. A defect slips through because a flaky test was dismissed as noise. Describe how you would redesign the team's response to red builds so this is less likely to recur, without simply slowing everything down.
Read the worked answer
The failure mode is social as much as technical: engineers learned that red usually means nothing, so they stopped investigating. The fix is to make red mean something again and to make investigation cheap.
- Triage by evidence, not vibes. When a test fails, the pipeline should automatically attach the failing test name, a link to its recent pass/fail history on the same commit, and whether it failed on a clean rerun of the same SHA. A test that fails twice on the same SHA is treated as a real defect until proven otherwise.
- Ownership. Every quarantined test has a named owner and a deadline. Unowned flakiness is the root cause of dismissal culture.
- Definition of done for red. A red build is either (a) fixed, (b) the test is quarantined with a ticket, or (c) the change is reverted. 'Rerun until green' is explicitly not an option for blocking tests.
- Blameless review. When a real defect is missed, run a short post-incident review that asks what made dismissal rational, not who clicked rerun. The answer here is that the signal-to-noise ratio made dismissal rational.
- Measure the ratio. Track false-failure rate as a first-class CI metric with a target (e.g., under 2%). Speed is preserved because the blocking suite stays fast and trustworthy; flaky tests move out of the critical path rather than slowing it.