Release Regression Triage Under a 36-Hour Deadline
A consumer payments product is 36 hours from a scheduled release. The automated regression suite contains 1,850 tests; over the last 30 runs, 7.4% of results flipped without a code change. On the release candidate, 112 tests failed: 61 are in modules untouched by this release, 23 match a known-flaky list, 18 sit in the payment-authorization path that changed, and 10 are environment timeouts on a shared staging cluster. Four testers are available, the staging environment is booked by another team for 12 hours, and code freeze starts in 12 hours. No performance test has been run on this build. The product owner wants a go/no-go recommendation by end of day. A customer-facing promotion launches two days after the planned release, and the rollback procedure has been rehearsed only once.
1. Build an evidence-based triage plan for the 112 failures. Which buckets do you act on first, and what must be true before a failure is waived?
Read the worked answer
Start by separating change-related signal from baseline noise, because the raw failure count is not a release metric. Map each failing test to the changed commits and to the last passing baseline; a failure in an untouched module that also failed on the previous build is evidence of an existing condition, not a new regression. Rerun the 10 timeouts once in a quiet environment with captured logs; if they pass and the logs show infrastructure contention, classify them as environment evidence, not product passes. For the 18 payment-authorization failures, require a deterministic reproduction or a clear test-data cause; because that path changed and carries revenue risk, these are the first action bucket and no waiver is acceptable without a named risk owner. For the 23 known-flaky tests, check their flip history in the last 30 runs; a quarantine label alone is not evidence. Waiver criteria: attributable non-product cause, no overlap with changed code, reproducibility evidence, compensating manual check, expiry, and owner sign-off.
2. Give the product owner a go/no-go recommendation with explicit conditions and a rollback trigger.
Read the worked answer
Recommend conditional go only if the payment-authorization failures are resolved or proven to be test defects, and if a smoke test of authorization, capture, refund, and settlement passes on the release candidate. The untouched-module failures should be compared with the prior baseline; if identical, record them as known and do not let them consume the 12-hour staging window. The 10 timeouts need one clean rerun, not repeated retries on the same noisy cluster. Because no performance test has run and a promotion follows in two days, add a short load smoke at expected promotion peak or explicitly accept that risk with a rollback trigger: authorization error rate above the agreed threshold, p95 latency above SLO, or duplicate charge reports. Tell the product owner the release is not 'green'; it is conditionally releasable with a narrow residual-risk statement, a rollback rehearsal owner, and a decision point after the payment evidence lands.
3. A team member proposes quarantining all 23 known-flaky tests and rerunning only the payment failures. What is the risk, and how would you control it?
Read the worked answer
Quarantining all 23 known-flaky tests and rerunning only payment failures is efficient but risky: flaky tests can be early indicators of real concurrency, timing, or data defects, and a blanket quarantine turns 'unknown' into 'pass.' The control is to quarantine by test ID with an owner, expiry date, and a requirement that any quarantined test covering a changed module be run manually or replaced by a targeted check. Track pass/fail history and require two consecutive clean runs or a root-cause fix before removal. Also keep the quarantined results visible in the release report so the product owner sees the coverage gap rather than a clean dashboard.