Kestrel: 1,200 UI tests and a cancelled exploratory session
Retail team Kestrel (six developers, two testers, two-week sprints, release candidate every Friday) runs 1,200 Selenium end-to-end tests in one browser suite. A full run takes 9 hours; 31% of runs fail for reasons unrelated to the change (timing, shared test data), and re-running only the failures takes another 90 minutes. Below that layer sit 140 unit tests and 60 API tests. Testers spend roughly 60% of their time repairing locators and resetting test data. Exploratory testing has shrunk to a single one-hour session per sprint, cancelled in four of the last six sprints. The engineering manager proposes two remedies: add automatic retries so the pipeline goes green, and hire a third tester to share maintenance. Two of the last four production incidents were business-rule errors in discount calculation.
1. Apply the test automation pyramid and the agile testing quadrants. What is wrong with the current distribution, and what target distribution would you argue for?
Read the worked answer
The shape is an inverted pyramid, an ice-cream cone: 1,200 end-to-end UI tests on top of 140 unit and 60 API tests.
- Economics: the cost to diagnose and repair a failure grows with the number of layers a test crosses. A UI check crosses browser, network, test data and application logic, which is why 31% of runs fail for reasons unrelated to the change. Feedback time grows with suite length; a 9-hour suite cannot run on every commit, so it is run rarely and its red builds are tolerated.
- Detection: discount calculation is a business rule. Such rules are cheapest to check at component or API level, where a failure means the rule is wrong. Under the current distribution a rule defect can only be seen through a slow, noisy UI path, and two of four incidents show that it is not being seen at all. Target distribution for this system: several thousand component checks running in minutes on every push; a few hundred API and service checks covering business rules and integration on every merge; roughly 80 to 100 UI end-to-end checks reserved for critical user journeys that genuinely span the stack, run per release candidate and nightly. Quadrants: the 1,200 UI checks are quadrant 2, but they are policing behaviour that quadrant 1 checks could own. Quadrant 3 has collapsed to a cancelled hour, and quadrant 4, performance and security, is not visible in the setup at all. A test that passes only on the third attempt is a defect in the test, not a pass; retries hide exactly the signal the team needs.
2. Propose a three-sprint rebalancing plan with concrete targets, and respond to the engineering manager's two proposals.
Read the worked answer
- Sprint 1: freeze new UI tests. Instrument the suite to record infrastructure failures and cluster them. Quarantine or delete the worst 200 checks, the ones whose failure never corresponds to a real defect. Build the API layer by converting the 150 highest-value UI checks into API-level checks; target 250 API checks. Success: runtime under 4 hours, flake rate under 10%.
- Sprint 2: push business rules down to component level, pairing a developer with a tester on discount calculation and the other rule-heavy areas currently covered only through the UI. Target 60% of remaining UI checks to have an equivalent lower-level check, then retire the duplicate. Restore exploratory testing: two 90-minute chartered sessions per sprint with a written debrief, tracked by charters planned, executed, completed and the risks found.
- Sprint 3: keep 80 to 100 UI checks for critical journeys, run nightly and per release candidate. Publish a scorecard: runtime, flake rate, escaped defects, automation coverage of the risk list, exploratory charters executed. On the two proposals. Automatic retries are not a fix: they conceal flakiness and train the team to ignore red builds. A single automatic re-run with mandatory triage is acceptable as a diagnostic, but only with the reason recorded. A third maintenance tester scales the wrong layer; at 1,200 brittle UI checks another person simply absorbs more repair and legitimises the anti-pattern. A hire is defensible once the suite is rebalanced, and then for exploratory and API-level skill rather than for locator repair.