1,850 UI Tests, Four Defects, and a 90% Target
Halcyon Travel has 1,850 automated UI tests for its booking site, run overnight. The nightly suite takes 9 hours on four shared agents and finishes after the morning standup. Twenty-two percent of tests fail intermittently, and the team routinely re-runs the suite to get a green result. Two of the four agents run an older browser version. Test data is created by hand for each run and reset only when someone remembers; three teams share the staging environment, which is refreshed from a production copy once a month. In the last six months the automated suite raised 4 defects, all of which developers had already found before the nightly run, while exploratory and manual testing raised 31, including the three highest-severity issues. Automation maintenance consumes about 60% of each sprint's test capacity. The engineering director has announced a target of '90% automation coverage and no manual regression within two quarters', and a commercial tool's licence, renewable in six weeks, is up for review.
1. Evaluate the automation effort against the director's target using the evidence. What does '90% automation coverage' fail to capture?
Read the worked answer
The evidence says the current effort is returning very little.
- 1,850 UI tests producing 4 defects in six months, all pre-empted by developers, means the suite is not the mechanism by which quality is being found.
- Exploratory and manual testing found 31 defects and the three highest-severity ones. Automation is not substituting for that; it is running alongside it at high cost.
- 22% intermittent failure is 407 tests whose results cannot be trusted. Re-running until green converts a signal problem into a data-integrity problem: the team no longer knows which failures are real, and a genuine regression can be buried in a re-run.
- 60% of sprint capacity on maintenance means the suite has become an obligation rather than an asset.
What '90% automation coverage' fails to capture. It measures how much of the existing test material has been automated - a supply-side metric that says nothing about risk coverage, defect detection, stability, runtime, or cost. Chasing it will, on this evidence, push the team to automate easy and stable cases (raising the number) while leaving high-risk, judgement-heavy areas untested and worsening the maintenance burden. It also sets an unreachable end state: 'no manual regression' is incompatible with the exploratory testing that found most of the real defects here.
Better measures: defect detection per execution hour, intermittent-failure rate, suite duration and time to result, maintenance hours as a share of capacity, and coverage of each risk area by automated checks.
2. Propose the changes you would make to the suite and its process over the next quarter, with priorities and expected effects.
Read the worked answer
Priority 1 - make results trustworthy. Quarantine all 407 intermittently failing tests, with a hard rule that re-running to green is banned. Classify each cause: hard-coded waits and timing, shared test data, environment contention, stale browser versions. Fix, rewrite at a lower level, or delete. Expected effect: a pass or fail that the team can act on, which is the precondition for everything else.
Priority 2 - cut the runtime and the scope. Nine hours is a symptom of testing too much through the UI. Move business-logic verification to API or service-level tests, keep UI tests for genuine end-to-end journeys and smoke. Target a suite that completes in under 60 to 90 minutes so results are available before the standup. Expect to remove several hundred tests without losing risk coverage.
Priority 3 - test data and environment. Generate data programmatically per run, with unique identifiers so runs do not collide, whether parallel or serial. Pin identical browser versions across all four agents. Agree a schedule and booking mechanism for the shared staging environment, and push for a pre-release environment where this team controls refresh timing.
Priority 4 - select by risk, not by coverage percentage. Automate stable, high-frequency, high-risk regression checks and smoke. Keep exploratory, usability, and new-feature investigation manual. Publish the selection rule so the director's target can be renegotiated on evidence.
Reporting. Monthly: intermittent-failure count, pass rate excluding quarantined tests, suite duration, maintenance hours, defects first detected by automation.
3. Recommend a decision on the tool licence and set out the criteria and metrics you would use to make it.
Read the worked answer
Recommendation: do not renew by default, and do not sign a multi-year term. Six weeks is enough to run a bounded evaluation, and the architecture is about to change, so locking in now would fix the tool choice before the strategy is settled. If the vendor offers a short extension or monthly term, take it and decide on evidence.
Criteria.
- Fit: does it support the browsers, agents, and CI pipeline actually in use, including parallel execution and reliable reporting?
- Stability and diagnostics: how well does it handle waits, retries, and failure artefacts such as screenshots and logs? Poor diagnostics is a direct driver of the 60% maintenance figure.
- Test asset portability: how expensive is it to leave? A tool that traps 1,850 tests in a proprietary format holds the team hostage at every renewal.
- Total cost: licence plus training plus maintenance hours, compared with the engineering hours it saves.
- Support and upgrade path from the vendor - including whether the older-browser problem is a tool constraint or an environment one.
Evaluation design. Four weeks, a representative subset of tests (a smoke pack plus one journey set) built in the incumbent tool and in one open-source alternative. Measure authoring time per test, failure stability across 20 consecutive runs, mean time to diagnose a failure, and maintenance effort.
Decision rule set in advance. Renew only if the incumbent shows a clear advantage on stability or authoring speed that survives the total-cost comparison; otherwise migrate incrementally, starting with the smoke pack, keeping the old suite running until the new one is proven. The one thing I would not do is let a renewal deadline make the decision.