Parallelism exposes non-determinism in a 1,240-test Selenium regression suite

Parallelism did not create the defects; it removed the accidental ordering and delay that were hiding non-determinism. Three distinct causes map to the three failure clusters.

1. Shared seeded customer. Tests logging in as the same customer contend for the same records: sessions invalidate each other, carts and locks collide, and one test's cleanup deletes another's data. Fix: per-test data provisioning through the application's own APIs, generating a unique customer per test with a reserved identifier range, plus teardown owned by the test. This is the highest-value change because it removes cross-test coupling rather than masking it.

2. Dashboard counter from an asynchronous job. Assertions here are effectively time-based; the test checks a value before the aggregation job has committed. Fix: replace sleeps with condition-based polling on a business-meaningful signal (job status endpoint, or an event published when aggregation completes), with a bounded timeout and a clear failure message. Polling alone is insufficient if the job is genuinely eventual; the test must define the freshness guarantee it is asserting.

3. Order then back-office verification. Two systems propagate asynchronously, so the second assertion races the integration. Fix: either assert at the integration boundary via a stub or queue inspection, or poll the back-office read model with a documented SLA. Do not simply raise the timeout, which converts a fast flake into a slow flake.

Beyond these, the suite needs independence as an explicit design rule: any test that cannot run alone is a defect in the test asset. Sequence-dependent chains should be refactored or moved to a small serial lane that is excluded from the gate.

Adding agents made contention worse and increased cost while leaving the root causes intact. Expected result after changes 1 and 2: flake rate under 2% and a 30-minute gate becomes credible.

The current 17% figure is a single aggregate that cannot guide action, and re-run-to-green hides the very signal needed. The approach:

Classify every failure at first occurrence. Triage categories: product defect, automation defect (script/logic), test data defect, environment/infrastructure defect, and unknown. The unknown bucket must trend toward zero; a large unknown bucket means triage is guesswork. Record the category and a root-cause tag (e.g. shared data, timing, selector, stub mismatch) in the test management or defect tool, not in a spreadsheet that goes stale.

Measure first-run results, not final results. Report pass rate on first attempt, and report the re-run pass rate separately. A test that passes only on re-run is counted as failed for reliability purposes even if the build is ultimately green.

Establish a stability baseline. Execute the full suite against an unchanged build ten times (or use the nightly history over two weeks) and compute per-test flake probability. Publish the distribution: how many tests are above 1%, above 5%. This turns opinion into a ranked repair backlog and lets you track the tail shrinking.

Track leading indicators alongside outcome metrics: count of tests containing fixed sleeps, count of tests sharing a seeded identity, number of distinct test data provisioning paths, and triage effort in person-hours per week. The last one is the business case: two testers at roughly a full day each is a measurable cost that the CTO will recognise.

Finally, use a control chart of first-run pass rate per build so improvement is visible against normal variation, and agree a target with the product owner, for example under 2% flake with no test above 1% in the merge gate.

A 30-minute gate is achievable but not with all 1,240 tests, and the wrong answer is to fit everything by cutting waits. Recommend a tiered design:

• Pull-request smoke (about 150 tests, under 10 minutes): authentication, one happy path per critical journey, and fast API-level checks of changed areas. Purpose is fast rejection of obviously broken changes.
• Merge gate (about 400 tests, 20-25 minutes): risk-ranked tests covering the three clusters plus the highest-risk business rules, scaled across shards once data isolation exists. Only tests with a demonstrated flake probability below 0.5% are eligible.
• Nightly full suite (all 1,240) plus exploratory: catches depth regressions the gate omits.

The eligibility rule matters more than the count. A flaky gate teaches developers to ignore red, which is already happening elsewhere in the organisation; a smaller trustworthy gate preserves its authority. The trade-off is explicit: a subset gate accepts that some regressions reach the mainline and are caught nightly, so the nightly result must be owned, actioned within one working day, and any failure must block the release train rather than being re-run away.

Preconditions before switching on the gate: data isolation for the seeded-customer tests, deterministic synchronisation for the async assertions, and a quarantine mechanism with an owner and expiry so nobody 'fixes' the gate by disabling tests silently. If those are not in place, run the gate in shadow mode for two weeks, record what it would have blocked, and publish the false-block rate. If false blocks exceed roughly 2%, delay the switch rather than weaken the criteria.