A 420-Test Selenium Suite and a Twenty-Minute Commit Budget
An online retailer has 12 Scrum teams sharing one Selenium WebDriver suite of 420 end-to-end UI tests. The suite takes 38 minutes on four parallel agents and fails 22% of runs; triage shows 60% are timing or test-data issues, 25% are duplicate coverage, 15% are genuine defects. The CI pipeline budget is 20 minutes for the commit stage, with a nightly full run allowed. The system has a monolith, six microservices, and a React front end. Two test automation engineers support all teams. Product owners want faster feedback and fewer false alarms. A vendor proposes a record-and-playback tool to let manual testers create more UI tests. The engineering manager asks whether to buy more CI agents, rewrite all tests as API tests, or restructure the portfolio. Current API tests number 80 and unit tests 2,100.
1. Evaluate the three options the engineering manager proposes and recommend a course of action. Use the numbers in the setup to justify the recommendation.
Read the worked answer
Buy more agents would reduce the 38-minute wall time, perhaps close to 20 minutes, but it leaves the 22% failure rate and 25% duplication untouched. The commit stage would still be distrusted, and agents cost money without fixing maintainability.
Rewrite all 420 UI tests as API tests would be cheaper to run but would destroy legitimate UI journey and visual coverage. API tests also cannot validate the React front end end-to-end. Moving everything to one layer simply swaps one imbalance for another.
Restructure the portfolio is the strongest option. Current shape is roughly 2,100 unit, 80 API, 420 UI: the pyramid is inverted at the top. Recommended actions: delete the 25% duplicate UI tests (about 105), fix or quarantine the 60% timing/data failures, move suitable business-rule checks down to API or component level, and retain around 100-150 stable UI tests covering critical journeys and cross-browser risk. Target the commit stage under 20 minutes with fast unit and API tests, and run the broader UI set nightly. The vendor tool would likely increase UI tests and worsen the maintenance problem. Success metrics: commit-stage duration, flake rate below about 2%, and no loss of critical journey coverage.
2. Define selection criteria for deciding whether an existing UI test should be kept, converted to a lower-level test, or deleted, and outline a migration plan across 12 teams with only two automation engineers.
Read the worked answer
Criteria
- Risk/business criticality: keep UI coverage for revenue-critical journeys, authentication, and cross-browser or visual risks.
- Layer suitability: business rules and calculations are better tested through API or unit tests; navigation and rendering need UI tests.
- Stability and failure history: repeatedly flaky tests are candidates for quarantine, redesign, or deletion.
- Duplication: delete if the same risk is already covered at a lower, faster layer.
- Frequency of change: stable areas need less exhaustive UI regression.
Migration plan
- Inventory and tag all 420 tests by owner, risk, layer, flake history, and runtime.
- Remove the roughly 105 duplicates first; this is low-risk and immediate.
- Stabilise test data and waits for the remaining tests; introduce API-based data setup.
- Convert selected checks to API/component tests, running old and new in parallel for one or two sprints.
- Keep a curated UI smoke/journey set. Assign each team ownership of its tests; the two engineers act as coaches and reviewers, not a central bottleneck. Publish a definition of done for automation and review new tests against the criteria before merge.
3. How would you measure whether the restructuring is successful and govern automation across the 12 teams?
Read the worked answer
Measure outcomes, not test counts. Useful metrics: commit-stage duration against the 20-minute budget, suite flake rate, percentage of failures requiring no investigation, escaped defects in critical journeys, and coverage of identified business risks by layer. Track whether teams are adding lower-level tests versus more UI tests.
Governance: create a lightweight automation guild with one representative per team, shared page objects/API clients and test-data utilities, and mandatory review for new end-to-end tests. Publish a dashboard showing runtime, flake rate, and risky gaps. Review monthly with product owners, focusing on feedback speed and false alarms. Avoid rewarding raw automation numbers, which would encourage exactly the duplication that caused the current problem.