The Business Case for 1,200 Flaky UI Scripts
A payments team has 1,200 Selenium UI regression scripts, most created by record-and-replay in 2021. A full regression run takes 14 hours on a 6-node grid and is triggered twice per four-week sprint. Across the last ten runs, 34% of scripts failed at least once, and the team estimates 70% of those failures were caused by timing, shared test data or environment differences rather than product defects. Four automation engineers spend roughly 60% of each 20-working-day sprint maintaining scripts and triaging failures; triaging one failure takes about 6 minutes. Before automation, the same regression scope was executed manually by 6 testers in 5 working days. The product has moved to a microservice architecture with 14 services and an Angular front end, and the UI now changes in almost every sprint. Management asks for a business case: either 'automate more to get the payback' or 'abandon UI automation'.
1. Compute the current cost of the automated regression per sprint and compare it with the manual baseline. Is the investment paying back?
Read the worked answer
Per sprint (4 weeks, 20 working days):
- Script maintenance: 4 engineers x 12 days (60% of 20) = 48 person-days.
- Failure triage: 34% of 1,200 = 408 failing scripts per run; 408 x 6 min = 2,448 min = 40.8 hours = about 5.1 person-days per run; two runs per sprint = 10.2 person-days.
- Total labour: about 58 person-days per sprint.
Manual baseline: 6 testers x 5 days = 30 person-days for the same scope, with no flakiness and no maintenance burden (though slower feedback and human error).
So the automation currently costs about 28 person-days more per sprint than the manual process it replaced - roughly 1.4 full-time engineers of pure waste. The 14-hour overnight run is cheap; labour dominates.
The honest conclusion is that neither management option is correct. 'Automate more' multiplies the loss, since the per-script economics are negative. 'Abandon' throws away genuine value, because nightly unattended regression and the ability to test 14 services continuously are real capabilities that the manual process never had. What is broken is the portfolio and the architecture of the tests, not the idea of automation.
2. Propose a target test-automation architecture and estimate its effect.
Read the worked answer
Rebalance by level, following the test pyramid, then fix the sources of flakiness.
Portfolio:
- Keep about 120-150 end-to-end UI scripts, one per critical user journey (pay, refund, login, settlement report). These are the only tests that justify browser cost.
- Move about 700 checks down to API and contract level. With 14 services, consumer-driven contracts catch most integration breakage in seconds and are unaffected by Angular markup.
- Delete about 350 scripts that are duplicates, cover non-critical paths, or assert on volatile UI text. Deleting tests is a legitimate, funded activity.
Engineering fixes:
- Replace implicit sleeps with explicit condition-based waits; ban fixed sleeps in review.
- Create test data through APIs, per test, with unique identifiers; no shared accounts.
- Virtualise or containerise the 14 dependencies so a run cannot fail because a downstream stub changed.
- Quarantine: a flaky test is automatically removed from the gate, tagged, and must be fixed or deleted within one sprint.
Estimate: UI maintenance on 150 stable scripts, about 6 person-days; contract and API suite maintenance, about 8; triage with an expected flake rate below 5%, under 2. Total roughly 16-18 person-days per sprint, versus 30 manual and 58 today. Payback is within one or two sprints of the restructuring effort.
3. What criteria should decide whether a test case is automated at all?
Read the worked answer
Automate on economics and on the nature of the check, not on 'we should automate everything'.
Automate when most of these hold:
- The case runs often (every build, every sprint) - frequency is the main driver of return.
- The expected result is deterministic and machine-checkable; no human judgement of appearance or usability.
- The feature and its interface are reasonably stable; a test that must be rewritten each sprint has negative return.
- Setup data can be created and torn down programmatically.
- The case covers a high-risk or business-critical path, where a silent regression is expensive.
- The test fits a level where it is cheap: a validation rule belongs in an API or unit test, not behind a browser.
Do not automate:
- One-off or exploratory checks, and anything whose value is in a human noticing something unexpected.
- Usability, accessibility judgement and visual aesthetics (screenshot diffing helps with layout regression but still needs human sign-off).
- Cases whose expected value is undefined or contested - automate a settled oracle, not a debate.
- Highly volatile UI text and sequencing that changes every sprint; test the underlying API contract instead.
A practical gate: for each candidate, record the number of runs per year, minutes saved per run, and estimated maintenance hours per year. Automate only where the saving exceeds maintenance by a clear margin, and re-evaluate annually - the portfolio that paid back in 2021 is not automatically the one that pays back now.