Coverage Up, Escaped Defects Up

84% line coverage with a low mutation score means the tests execute code without checking it. Coverage is a necessary condition, not evidence of verification. The escaped defects are a better signal than the coverage number: they cluster at payment and tax integration boundaries, which the UI suite touches only through the browser and the unit tests mock away.

Why the numbers mislead:

• Line coverage rises when a test merely calls a method; assertions can be absent or trivial.
• A UI-heavy suite gives broad but shallow reach; it cannot isolate a tax-rounding or currency-conversion error.
• A 12% flake rate trains people to re-run until green, converting real failures into noise.

A balanced set:

• Mutation score on payment and tax modules, targeting 60 to 70%, because it measures assertion strength rather than execution.
• Contract tests for each external payment and tax interface, run on every build.
• Defect escape rate by component, with each escaped defect linked back to the missing test.
• Changed-code coverage rather than whole-repository coverage.
• Test runtime budget and flake rate as first-class quality metrics.

Reasoning: you get what you measure. If the only dashboard is coverage, the cheapest way to improve it is more assertion-free tests. Pair every coverage metric with a strength metric and a defect outcome, or it will be gamed within a quarter.

Rebalance, but replace before deleting. Deleting 1,200 UI tests first would trade a slow suite for blind spots, and adding unit tests to reach 90% repeats the same metric mistake at a different layer.

Safer sequence:

1. Classify the UI tests. Many are integration tests wearing a browser: form validation, calculation, permission checks. Those belong at API or component level, where they run in milliseconds and localize failure.
2. Build the replacement layer first for the highest-risk flows: payment authorization, tax calculation, refunds. Contract tests at the service boundary plus component tests for pricing rules.
3. Keep a small end-to-end smoke set, roughly 20 to 40 tests, covering critical user journeys in real browsers, run pre-release and on a schedule rather than on every commit.
4. Delete by traceability, not by count. A UI test can go when its scenario is covered at a lower level and the lower-level test has been shown to fail under mutation. Track the mapping so coverage of risk stays visible.
5. Target shape roughly 70% unit and component, 20% integration and contract, 10% end-to-end for this system, driven by the defect distribution rather than by a textbook pyramid.

What not to do: set a coverage target for the new tests, or the same Goodhart problem repeats. Measure mutation score on pricing and tax logic, and defect escapes, instead.

Treat flakiness as a defect with an owner and an SLA. At 12%, roughly one run in eight fails for reasons unrelated to the change; developers learn to ignore red builds, which is the real cost.

Policy:

• Detect automatically: run new and changed tests repeatedly in a nightly job and flag any non-deterministic result.
• Quarantine immediately: remove the test from the blocking set, keep running it in a non-blocking job, assign an owner, and set a five-working-day fix SLA. Quarantined tests appear on a dashboard; they are not silently deleted.
• Root-cause categories: clock and time, async waits, shared state and test order, test-data collisions, environment or network. Fix the cause rather than adding sleeps, which hide the race and slow the suite.
• Limit retries: one automatic retry may be acceptable for infrastructure noise, but retries must be counted and reported. Unlimited retry-until-green is how a real regression ships.
• Block merges on new flakiness, not on the inherited backlog, so the team can pay down the old problem without freezing delivery.

Metrics: flake rate per suite and per test, quarantine age, and re-run count. Target below 1% before the suite is trusted as a release gate.

Reasoning: a flaky suite is a measurement instrument that has lost calibration. Fixing it is cheaper than the review effort wasted on false failures, and it restores the signal that the escaped-defect analysis depends on.