Regression Suite Runtime vs Release Window
A payments team runs a 1,850-case automated regression suite that takes 9 hours on 6 parallel runners. The release train ships twice weekly, and the regression must finish within a 4-hour window before the Friday cut. Last month two defects escaped: a rounding bug in currency conversion and a broken retry on timeout. Analysis shows 62% of the suite covers features untouched in 14 months, while the conversion and retry modules have only 40 cases combined. Flaky tests fail 8% of runs and are usually rerun manually. The team lead proposes buying 10 more runners to fit the window. The QA manager wants to cut low-value cases and add targeted tests instead.
1. Evaluate the team lead's proposal against the QA manager's. Which addresses the actual risk, and what evidence would you require before deciding?
Read the worked answer
Adding runners fixes duration, not risk. The escapes were in conversion and retry logic, which have only 40 cases, while 62% of the suite guards stale features. More hardware makes the wrong tests faster: the 4-hour window closes, but the coverage gap on the changed modules remains. The QA manager's direction is better aligned with risk-based testing, but cutting 62% blindly is also unsafe.
Evidence needed: (1) change-frequency and defect-density data per module over the last two releases; (2) traceability from each of the 1,850 cases to a requirement or risk; (3) failure history, to confirm the stale cases never catch anything; (4) defect leakage by module, which points to where new tests are needed. Only then can you retire cases with justification rather than by age alone. Flakiness should be treated as a defect: an 8% failure rate means roughly 148 unreliable results per run, eroding trust in the suite. Fix or quarantine flaky tests before scaling out.
2. The manager wants the suite to fit 4 hours within one sprint. Draft a prioritization approach and state what you would not do.
Read the worked answer
Prioritize by risk: probability of failure (change frequency, complexity, past defects) times impact (financial, regulatory, customer-facing). Tier 1: conversion, retry, and any module changed this release, run on every commit. Tier 2: stable high-impact modules, run nightly. Tier 3: low-change, low-impact cases, run weekly or before major releases. Quarantine flaky tests into a separate job with an owner and a fix deadline.
What I would not do: delete cases without traceability evidence, and I would not assume parallel runners alone solve the window. I would also not accept a green suite as meaningful while 8% of runs are unreliable. Set a measurable exit: Tier 1 completes in under 45 minutes, total critical-path runtime under 4 hours, flake rate below 2%, and new tests for conversion and retry added and passing. Track escaped defects per release to confirm the change actually reduces risk, not just runtime.