Re-prioritising automation coverage against risk when 62% automated still leaves the risky areas manual
Halcyon Retail has 2,300 manual regression test cases for its order management system. 62% are automated, but a coverage analysis shows automation touches only 18% of the requirements rated high risk, while it heavily covers pricing screens that change twice a year. Manual regression takes nine working days per release, and releases ship every six weeks. Suite maintenance consumes about 35% of the four-person automation team's capacity, and the team spends a further 15% re-running tests that fail for environment reasons. Escape analysis from the last four releases shows 71% of production defects originated in three areas: payment authorisation, inventory reservation, and order cancellation. Two of those areas have no automated checks at all because 'the logic is too complex'. Management has approved budget for two more automation engineers and wants a plan for the next two quarters that shortens the manual regression window.
1. Propose a prioritisation model for what to automate next, and apply it to this data to state what you would do first, second and not at all.
Read the worked answer
The 62% figure is a vanity metric: it measures volume of cases, not risk covered. Prioritise on expected value rather than case count.
Model. For each candidate area score: business risk (impact of failure, from escape analysis and business owners), change frequency (how often the area is modified), manual execution cost (time per run x runs per year), stability/automatability (how deterministic and accessible the interface is), and automation cost (effort to build, including data and environment). Rank by (risk x frequency x manual cost) / (automatability-adjusted automation cost). The escape data is the strongest input available because it is observed, not estimated.
First: payment authorisation, inventory reservation and order cancellation. They account for 71% of escaped defects, so they dominate the value side. 'The logic is too complex' is an argument for better technique, not for exclusion: test these at API and service level with contract and state-based tests, use pairwise and boundary techniques over the decision tables, and virtualise the payment provider. Practical first step is a two-week spike per area to prove the interface supports automation, because if the only access is through a brittle UI, cost rises sharply.
Second: reduce the waste that starves capacity. The 35% maintenance plus 15% re-run overhead is half the team's capacity; fixing the environment-caused failures and the worst-maintained scripts buys more than a new hire. Then automate high-frequency, high-risk flows.
Not at all: pricing screens that change twice a year and are already covered, purely cosmetic checks, one-off regulatory reports, and cases that are cheaper to run manually a few times a year. Explicitly de-automate or retire low-value scripts to fund the new work.
Sequence the two quarters so that the first measurable benefit is a shorter manual regression window, not merely more automated cases.
2. How would you define and report the value of this programme so management sees progress before the manual regression window actually shrinks?
Read the worked answer
The manual regression window is nine days and will not fall until automated checks demonstrably replace manual ones in the risky areas, which is likely two to three quarters out. Reporting only that outcome would leave the programme invisible for months, so report a leading-to-lagging chain.
Coverage of risk, not of cases. Report the percentage of high-risk requirements with at least one automated check that has run green in the last 30 days. Baseline is 18%; a realistic target might be 45% in two quarters. This directly addresses the mismatch management should care about.
Escape trend. Defects escaping to production per release, tagged by area, with the three focus areas tracked separately. A reduction in payment authorisation escapes is the clearest evidence the programme works, though it lags by a release or two and must be normalised for release size.
Capacity released and capacity lost. Track manual regression person-days per release, and separately the maintenance and re-run overhead. If the team automates risky areas while overhead stays at 50%, the window will not shorten; showing both prevents the team from being congratulated while treading water.
Feedback quality. Time from commit to a trustworthy verdict in the automated pack, and the flake rate on first run. A fast pack that cannot be trusted has no value.
Leading indicators for the new engineers: ramp time to first merged automated test, number of areas with API-level test support, and number of tests retired. Include one honest counter-metric: manual cases not yet automated in high-risk areas, so the remaining gap stays visible rather than being buried in a rising overall percentage.
3. What would you do about the two areas described as 'too complex to automate'? Give an approach and state the risks in it.
Read the worked answer
Treat 'too complex' as a diagnosis to be tested, not a verdict. In practice it usually means the interface is awkward, the logic is combinatorial, or nobody has modelled the rules.
Step 1: find the access path. Determine whether the logic is reachable below the UI: a service endpoint, a rules engine with an exposed decision API, or a database state change observable through a query. Service-level testing is far more tractable than driving the UI, and failures are diagnosed faster. If there is genuinely no seam, that is an architecture finding to raise with development, not a permanent testing excuse.
Step 2: model the rules. Extract decision tables for payment authorisation and cancellation from the specification, then apply equivalence partitioning, boundary values and pairwise combination to the condition set. Complexity in the sense of many rules is exactly what technique handles; complexity in the sense of non-deterministic third parties is what virtualisation handles.
Step 3: virtualise the uncontrolled parts. Stub the payment provider with responses driven by a contract, and test the cancellation flow against deterministic time and message ordering. Keep a small number of end-to-end tests against the real provider in a nightly lane so the stubs do not drift.
Risks. Stubs can encode the team's misunderstanding of the provider rather than its real behaviour, so contract tests must run against the live interface regularly. Service-level tests can miss UI-level defects such as mis-mapped fields, so retain a slim end-to-end path. The decision-table work is skilled and slow, so budget a spike with a named owner and an explicit exit criterion: if a viable seam and a stable stub cannot be demonstrated in two weeks, escalate as a risk with the business impact of continued manual-only coverage rather than quietly dropping the area.