Making a quality gate and dashboard meaningful when the pipeline is green and production is not

The dashboard measures activity, not confidence. A 96% pass rate across 3,900 tests is dominated by tests that never fail and says nothing about the pricing rules that actually broke. Meanwhile a 210-test quarantine list, including the only refunds coverage, is a silent coverage hole that the headline number disguises. The green board is therefore anti-informative: it reassures while the riskiest behaviour is untested.

Replace it with a small set tied to decisions:

• Escape rate and detection stage. Defects found in production per release, normalised by change volume, and the stage at which defects are caught. This is the outcome metric.
• Change risk coverage. For each pull request touching a business rule, was there an automated check covering that rule? Report the proportion of high-risk changes with a matching automated check; this would have exposed the pricing gap directly.
• Suite trustworthiness. First-run pass rate, flake rate, and number of tests quarantined with age. Trust is a precondition for a gate.
• Feedback time distribution. Not an average: report the 50th and 90th percentile time from pull request to verdict, against the 20-minute budget, because the average hides the long tail that drives developers to merge on red.
• Quarantine debt. Count and age of quarantined tests, with the risk they cover, so debt is visible and prioritised rather than buried.

Each metric needs an owner and a threshold that triggers action. A monitor with no agreed response is decoration, and developers already demonstrate what happens when the board is not believed.

A single blocking gate over everything would exceed the 20-minute budget and be ignored, reproducing today's behaviour. Use layered gates with explicit contracts.

Stage 1, pull request, target under 20 minutes: unit tests plus a risk-selected API subset and contract tests. Selection is by change impact (which services and rules the diff touches) rather than by test count, backed by a nightly full API run so selection does not become a coverage gap. Blocking: yes.

Stage 2, merge to main, target under 45 minutes: full 3,000 API tests plus a curated UI pack of the highest-risk journeys. The 55-minute nightly UI suite cannot be moved here wholesale; split it so the critical journeys run per merge and the remainder stay nightly.

Stage 3, nightly and pre-release: full UI suite, real-interface end-to-end lane, exploratory and non-functional checks.

Eligibility rules for any blocking stage: no test with a flake rate above 0.5% over the last 30 runs, no test in quarantine, and every test traceable to a requirement or risk. A test that does not meet these is excluded and logged, not silently tolerated.

Quarantine policy: maximum 14 days, mandatory owner and ticket, and the risk covered recorded. New quarantines beyond a threshold per month trigger a stop-and-fix review. Refunds coverage cannot be quarantined without a release manager exception.

On failure: the gate blocks deployment, the failure is triaged within one working hour, and overriding requires a named approver with a recorded reason. Re-running until green is prohibited; a re-run is only valid after a diagnosed cause is recorded.

A weekly pattern is a strong clue that this is not product instability but a scheduled interaction. The team's reaction, merging on red, is a rational response to an untrustworthy signal and must be addressed alongside the cause.

Diagnose from data, not anecdote. Extract the last 8-12 weeks of API failures and correlate by day, hour, test and environment. Weekly patterns typically come from: a nightly or weekly data refresh or purge job that changes seeded records; environment maintenance windows or certificate and token expiry cycles; shared environment contention with a batch process; a scheduled third-party sandbox reset; or scheduled security scanning that throttles the environment. The correlation should identify the specific tests and the specific dependency, and checking the platform team's change calendar in the same week usually confirms it.

Fix by independence, not by calendar. Move the tests off the mutable dependency: provision their own data, stub or virtualise the affected third party, or isolate a stable environment for the blocking stage. If a shared dependency cannot be removed, the gate tests must not depend on it; move those to the nightly lane and record the residual risk explicitly rather than excusing red.

Re-establish the signal's credibility. While the cause is open, put an automated detector on the weekly failure so it is attributed to the environment rather than the product, and stop the practice of merging on red by making any override an explicit, recorded, time-boxed decision with a name attached. Add a review at the next retrospective of how many merges occurred on red and what was in them; if any defect escaped through such a merge, the gate policy needs tightening, not loosening.