1,200 UI Tests and a Request for More Agents
Vertex Banking has 1,200 Selenium UI regression tests covering its online account application. A full run takes 9 hours on two agents. Last month 68% passed, 12% failed with genuine defects and 20% failed intermittently; the team reruns failures up to three times and treats a pass on rerun as a pass. Two testers spend roughly 40% of their week maintaining locators and test data. Test data is created by running a 30-minute SQL script, and accounts are shared across tests. There are no API-level tests. The CI pipeline runs the suite nightly; developers receive results at 9 a.m. the next day. The delivery manager wants to double release frequency and asks whether to buy two more agents to run the suite faster.
1. Respond to the delivery manager's proposal: is buying two more agents the right investment? Justify with an alternative architecture.
Read the worked answer
No. Two more agents reduce wall-clock time, but they leave the 20% flakiness and the 40% maintenance load untouched; you would get faster, still-untrustworthy answers. The 9-hour run is not the main constraint - feedback arriving at 9 a.m. the next day is.
Rebalance toward a layered strategy. Move business rules and service behaviour to API-level tests, which are faster and more stable than UI scripts. Add consumer-driven contract tests where services integrate. Keep UI automation for a small set of critical end-to-end journeys that genuinely need a browser. Create test data through APIs with isolated accounts per test instead of shared accounts and a 30-minute SQL script.
Parallelisation can follow stabilisation, not replace it. If the suite is not trustworthy, more hardware only scales waste. Present the comparison as cost per reliable signal and time to feedback, not cost per agent.
2. Propose how to handle the 20% intermittent failures.
Read the worked answer
Classify before fixing. Typical causes are asynchronous waits, shared test data collisions, test-order dependence and environment instability. Quarantine intermittent tests so they stop polluting the pass rate, assign an owner and set a fix-or-delete deadline.
Stop treating a rerun pass as a pass. It hides a race, a data leak or a genuine intermittent product defect, and it trains the team to ignore red results. Track the flaky rate and the mean time to fix, and review them alongside genuine failures.
Fix root causes in the framework: explicit waits for conditions rather than sleeps, independent test data per test, deterministic setup and teardown, and isolation from other tests. Delete tests that no longer earn their keep. Until the flaky rate is low, expanding coverage or adding agents increases noise rather than confidence.
3. How would you measure and report the value of the automation effort to the delivery manager?
Read the worked answer
Report a balanced set tied to the manager's goal of releasing more often with confidence: time from commit to trustworthy result, valid pass rate excluding quarantined tests, flaky rate, maintenance hours per week, and coverage of the highest-risk areas rather than a raw test count.
Add defect-oriented measures: defects found by automated checks before release, production escapes in automated areas, and cost per run. Compare against the manual regression effort the automation replaces. Establish a baseline before changing the architecture and review monthly so the trend is visible.
Avoid vanity metrics such as number of test cases or lines of code. A large suite with low trust is a liability. The message to the delivery manager should be that the investment is justified by faster feedback and lower risk exposure, and that this requires rebalancing the suite, not just buying hardware.