Thirty-One Percent Flaky, Three Retries, and a Request for More Agents
Vertex Health's clinical portal team keeps 9,800 automated checks in one repository: 7,600 Selenium UI tests, 1,400 API tests and 800 unit tests. The suite takes 3 hours 40 minutes on a 40-agent grid, so it runs nightly plus a 900-test smoke subset per pull request. Thirty-one percent of nightly runs fail, and re-running the same commit unchanged passes 84% of the time. The page-object layer is 24,000 lines, and 60% of it was written by two contractors who have since left. Test data comes from a nightly production clone with masking, which takes five hours; parallel workers collide on the same patient identifiers, and 11% of failures are duplicate-key errors. The team's response has been to allow three attempts per test. The engineering director wants the suite on every pull request within 20 minutes and believes the answer is more grid agents.
1. Assess the director's diagnosis. Would more grid agents deliver a 20-minute pull-request suite?
Read the worked answer
Throughput is not the binding constraint. With 40 agents and 9,800 tests, the suite averages about 54 seconds of wall time per test, which is far too long for tests that should be milliseconds to a few seconds. That indicates waiting, sleeps and environment contention inside the tests, not a shortage of compute. Adding agents parallelises the waiting.
The failure rate dominates the cost. With 31% of nightly runs failing and 84% passing on unchanged rerun, the majority of failures are non-deterministic. More agents increase contention on shared data and typically raise the flake rate, so the director's fix could make the reported symptom worse. The three-attempt retry policy compounds this: it hides roughly a third of the noise while tripling runtime for the affected tests and destroying the team's ability to see flakiness at all.
The arithmetic for 20 minutes. To run 9,800 tests in 20 minutes on the same grid you would need each test to average a few seconds and near-perfect parallel efficiency. The realistic path is to run far fewer tests per pull request: a selected subset of unit and API tests plus smoke journeys, which is achievable today once the flake and data problems are fixed.
Order of work. Remove retries to make flakiness visible; fix test data isolation; quarantine the worst offenders; push assertions down from UI to API and unit; then revisit infrastructure. Compute spend is the last lever, not the first.
2. Write the flaky-test policy: detection threshold, quarantine mechanics, ownership, and exit criteria.
Read the worked answer
Detection. A test is flagged when it fails and then passes on an unchanged commit, or when it fails on one agent and passes on another in the same run. Because retries are being removed, this signal becomes reliable. Record the flag automatically with the test ID, branch, timestamp and failure signature; no human triage is needed to raise it.
Quarantine. Two flags within 14 days triggers automatic quarantine. Quarantined tests are removed from the blocking gate but continue running in a non-blocking job, tagged so their results never count toward a coverage or pass-rate claim. The suite has a quarantine budget, for example 2% of total tests; exceeding it is a release-quality escalation, not a routine statistic.
Ownership. The owning squad, not a central automation team, owns the fix. The test is attached to the feature area it covers, and the squad's lead is named automatically. A shared automation guild may help, but ownership stays with the squad that would suffer the regression.
Exit criteria. Twenty consecutive passes across at least three branches and two agent types, or the test is deleted and its intent re-expressed at a lower layer within ten working days.
Anti-gaming rule. Quarantine is not a place to park difficult tests permanently. Age of quarantined tests is reported weekly alongside the count, and anything over 30 days is deleted by default.
3. Test data is a major contributor. What would you change, and in what order relative to reducing the UI suite?
Read the worked answer
Fix data first, and narrowly. The 11% duplicate-key failures are a data isolation defect, not a flakiness mystery: parallel workers are writing to the same seeded patient records. The cheapest fix is per-worker data namespacing, generating identifiers with a worker prefix and a run ID, plus a teardown that deletes by that prefix. That removes a large share of the 31% failure rate for a small engineering cost, and it makes every later measurement trustworthy. Do this in week one.
Then stop depending on a production clone for most tests. A five-hour nightly clone enforces a stale, shared, non-deterministic dataset. Move unit and API tests to on-demand synthetic data created by a factory in the test process itself. Reserve the masked clone for a small number of exploratory and pre-production journeys that genuinely need production-like volume and distribution.
Then reduce the UI suite. Do it after data isolation, because at present you cannot tell a genuine UI failure from a data collision, so any pruning decision is guesswork. Once the noise floor drops, cluster the 7,600 UI tests by journey, keep a representative subset, and re-express the rest at API or unit level.
Also address maintainability. Twenty-four thousand lines of page objects written mostly by two departed contractors is a documentation and abstraction problem. Introduce a thin, team-owned layer and delete abstractions nobody uses, rather than rewriting the whole layer at once.