Rescuing an oversized, flaky UI automation suite

The BDD rewrite does not address any of the stated problems. Cucumber is a specification and collaboration layer; it sits on top of the same Selenium calls, the same shared data and the same environment, so the 7.5-hour runtime and 28% flakiness would survive the rewrite intact - at the cost of a large migration and 400 more slow UI tests. Rewriting everything is also the most expensive possible response when the suite has never been triaged.

A target architecture should be layered:

• API/service layer for business rules, calculations and, importantly, test data setup. Policy records should be created through the API or a data service, not by driving screens.
• UI layer reserved for journeys that only the UI can verify: rendering, navigation, client-side validation, session and security behaviour. This should be a small, stable set.
• A keyword or screenplay-style abstraction layer so tests describe intent rather than locators, reducing maintenance when the UI changes.
• Test data management with unique data per test, generated programmatically, plus environment provisioning controlled by the pipeline.
• Reporting and CI orchestration that separate a fast pull-request gate (API plus a UI smoke subset) from the full nightly regression.

On the product owner's question: the UI suite cannot simply be replaced - API tests cannot verify rendering or client-side behaviour - but the large majority of the 1,150 tests probably can be re-expressed as API tests or retired. That decision needs the portfolio analysis described below, not a wholesale swap.

Flakiness at that rate is almost always environmental and structural rather than a Selenium quirk. The setup names most of the causes:

1. Shared environment. QA uses the same environment during the day, so data and state change under the suite. Fix: a dedicated, reproducibly provisioned environment for automation, or at minimum a booking mechanism and a reset to a known state.
2. Shared test data. Every test operating on one seeded policy creates order dependencies and collisions, especially when tests run in parallel. Fix: each test creates its own uniquely identifiable data through the API and cleans up (or uses disposable tenants).
3. UI-based data setup. This is slow and fragile: any UI change breaks setup for many tests, and a mid-test failure leaves orphaned data. Fix: move setup to the API or database layer.
4. Synchronisation. Fixed sleeps or implicit waits produce failures under load and pass locally. Fix: explicit waits on observable conditions, with timeouts, and no arbitrary sleeps.
5. Test independence. Tests that depend on order cannot be parallelised or rerun reliably. Fix: enforce independence in review, and run tests in randomised order to expose hidden coupling.

Process: quarantine newly flaky tests out of the gate, triage them by root cause, and track a flakiness rate as a first-class metric with a threshold. Rerunning failures to "get green" hides the problem and destroys trust in the results.

The decision must be evidence-based per test, not a blanket policy. For each of the 1,150 tests, collect value and cost: defects it has found, business criticality of the behaviour, execution time, failure rate, and how often the covered area changes. Then:

• Keep and stabilise: high-value UI-specific tests (login, policy issuance journey, client-side validation) - a few dozen, not hundreds.
• Rewrite at API level: tests that verify business rules through the UI only because that was the available interface. This is likely the largest group and gives the biggest speed and stability win.
• Retire: duplicates, tests of behaviour no longer present, and tests of low-risk areas that add no defect-detection value.
• Hold as manual/exploratory: usability, visual judgement, complex one-off scenarios where automation cost exceeds benefit - automation should not be assumed for everything.
• Add new coverage only after triage; the 400 proposed claims tests should be written at the cheapest effective layer, with UI used sparingly.

Pipeline split: on pull request, run API-level tests plus a curated UI smoke set - a handful of critical journeys - with parallel execution, targeting under 30 minutes; run the broader regression nightly and before release; publish pass rate, duration and flakiness trends so the investment can be justified. Also define the automation ROI measures up front (feedback time, escaped defects in automated areas, maintenance hours per release) so the 30-minute target is treated as an engineering constraint rather than a wish.