The Flaky Test That Everyone Learned to Ignore
A team's CI pipeline runs 1,800 tests. About 60 fail intermittently, roughly 3% of runs. Engineers have adopted a habit: if a red build touches only those 60, they re-run the pipeline until it goes green. Median time to green is now 25 minutes. Last month a real defect reached production because it was masked by a re-run. The QA lead proposes quarantining the 60 flaky tests into a non-blocking job. A senior engineer objects that quarantine is where tests go to die and the team will never fix them. The engineering manager wants the pipeline green and fast, and asks for a decision this week.
1. Diagnose the actual quality problem here, and explain why 'green and fast' is the wrong framing.
Read the worked answer
The pipeline is already telling the truth; the team has learned to disbelieve it. The 3% flake rate is not the core problem - the re-run habit is, because it converts a signal into noise and destroys the pipeline's authority. The production escape is the predictable consequence: once 'red' can mean 'probably flaky', every red is suspect, including the real one.
'Green and fast' optimizes the wrong variable. A fast pipeline that is not trusted is worse than a slower one that is, because engineers will route around it. The right framing is signal integrity: a red build must mean 'something is wrong' with high confidence, and the time cost of investigating a red must be low. Quarantine and re-run both attack the symptom; only one of them preserves the signal, and only if quarantine is paired with ownership.
2. Evaluate the quarantine proposal against the senior engineer's objection, and specify the conditions under which quarantine is acceptable.
Read the worked answer
The objection is historically correct: unowned quarantine becomes a graveyard. But the alternative on the table - keep re-running - is strictly worse, because it actively erodes trust. Quarantine is acceptable if and only if it is a bounded, owned process, not a dumping ground:
- Each quarantined test gets an owner and a ticket with a deadline (e.g. 30 days).
- The quarantine job still runs and reports, so flakes are visible rather than silent.
- A test that is not fixed or deleted by its deadline blocks the owning team's next release, not the whole pipeline.
- New tests cannot enter quarantine without a ticket created at the same time. Under those rules, quarantine is a triage tool with an exit; without them, the senior engineer is right and the team is just hiding the problem.
3. The manager wants a decision this week. What do you decide, and what do you commit to measuring?
Read the worked answer
Decide: approve quarantine with the ownership rules above, and simultaneously ban blind re-runs - a red build must be triaged, not re-run, unless the failure is on the quarantine list. That single rule restores meaning to red immediately and costs nothing.
Commit to measuring: (1) flake rate in the blocking job (target near zero), (2) number of quarantined tests and age distribution, (3) median time from red to diagnosis, and (4) count of production escapes attributable to masked failures. Review at 30 days. If the quarantine count is growing and ages are climbing, the process has failed and the manager should escalate to a stop-the-line fix; if ages are falling, the objection is answered with data rather than argument.