The 82% Coverage That Ships Bugs

The proposal optimises the metric that is already passing while the metric that is failing goes untouched. Line coverage is 82% and nine defects escaped. If low coverage were the binding constraint, the escapes would be scattered across under-covered code. Instead seven of nine cluster in two areas - webhook retries and timezone boundaries. That is the signature of a portfolio gap, not a volume gap: the team is not exercising duplicate deliveries, out-of-order deliveries, DST transitions, and cross-midnight boundaries, and no number of additional unit tests on already-covered invoice-total arithmetic will find those.

Specific problems with adding 400 unit tests to reach 90%:

• It inflates a number that does not predict escapes. Line coverage measures execution, not verification. A test that calls a method and asserts nothing raises coverage; a test with a weak oracle passes a bug.
• It targets the wrong layer. Retry and timezone defects are interaction and integration defects. They manifest when the provider sends the same webhook twice or when a clock crosses a boundary - conditions that unit tests with mocked clocks and mocked HTTP clients are structurally unlikely to surface, and that developers tend to mock away.
• It costs real time. 400 tests at even four seconds of runtime each adds roughly 27 minutes to a 52-minute pipeline. A slower pipeline with a 26% flake rate and a retry culture makes engineers trust the signal less, not more.
• It invites gaming. When a gate becomes a target, teams write assertion-free tests to clear it. Goodhart's law is not a theoretical concern here; it is the predictable response to a hard threshold.

The deeper problem the proposal does not name is that the team has lost confidence in its own signal. Two regressions merged after reruns went green means engineers no longer read failures. A test suite nobody reads is not a quality gate; it is a ritual.

Framing for the sponsor: 'Coverage tells us how much code the tests touch. It does not tell us whether they would catch the bugs we actually shipped. Nine Severity-2 escapes at 82% coverage is the proof. I am proposing to keep coverage as a floor and make escaped defects the number we manage.' That respects the sponsor's instinct - they want fewer production defects, and the 90% figure was a proxy for that - while redirecting the effort.

Risk-based portfolio, targeted at the two clusters:

1. Consumer-driven contract tests for the payment provider. Record real provider payloads as fixtures; verify our consumer expectations on every build; run against the provider sandbox nightly. A contract test catches a changed field name before production, which unit tests cannot.
2. Provider-double integration tests with adversarial delivery. Duplicate webhook, out-of-order webhook, delayed webhook beyond the retry window, webhook delivered after the payment was refunded. Assert idempotency: the same event ID twice produces one ledger entry. These are the exact conditions in seven of nine escapes.
3. Time-boundary tests, preferably property-based. Invariants such as 'due date equals issue date plus payment terms, evaluated in the tenant's timezone' and 'an invoice is never dated before its issue instant'. Generate cases across DST start and end, month ends, leap day, and 23:00 local. A property-based generator finds the boundary cases humans forget.
4. Mutation testing on the two high-risk modules only. Mutation score, not line coverage, measures whether assertions have teeth. Scoped to two modules it is affordable; run nightly, not on every commit.
5. Coverage as a floor: keep the gate at 80% as a floor that blocks nothing but a regression, and stop reporting it as the headline quality number.

Metrics to report instead: escaped-defect count per release (target: down from 4.5 to under 1), defect-detection percentage (share of defects found before production), flake rate, and pipeline duration. These move when the portfolio improves; coverage does not.

Treat flakiness as a defect with an owner and a deadline, and make retries visible. The retry culture exists because the pipeline rewards green and nobody is accountable for a red that goes green. Both have to change at once.

1. Quarantine, do not block. Move flaky tests to a separate non-blocking job that still runs and still reports. Delivery keeps moving; the signal is preserved.
2. Assign ownership. Every quarantined test gets a named owner, a ticket, and a 10-business-day budget. At the deadline it is fixed or deleted. A deleted flaky test is honest; a permanently quarantined test is a slow leak. A test that has been flaky for a quarter is not protecting anything.
3. Make reruns countable. The pipeline records attempts per test and publishes a flaky rate; retries must be annotated, not silent. If a job is retried, the CI dashboard shows it and the owning team gets a notification. Two regressions merged after silent green reruns is precisely the failure mode this prevents.
4. Categorise root causes rather than fixing symptom by symptom. The usual suspects in a 210-test integration suite: unmanaged time (fixed sleeps, real clocks), shared mutable state between tests (one database, no per-test schema), ordering dependencies, and unawaited asynchronous assertions. Fix them at the harness level - injectable clock, per-test schema or transactional rollback, deterministic awaits on conditions rather than sleeps, seeded randomness - and a large fraction of the 26% disappears with a handful of changes.
5. Protect the merge gate. Retries are permitted only on the non-blocking job. The blocking gate reports first-attempt results, so a flaky test cannot mask a real regression.
6. Report the trend alongside escaped defects, so improvement is visible and the sponsor sees the same discipline applied to reliability as to coverage.