6,100 Browser Tests and a Skip Label

The portfolio is inverted

6,100 end-to-end tests is roughly thirty times a healthy count for a system this size, while 900 unit tests across 220,000 lines is thin. The cheapest, fastest layer is the smallest and the most expensive, most brittle layer is the primary safety net. That is the classic inverted pyramid, and its symptoms are all present: a 4-hour blocking suite, 11% flake, and a bypass label used on 38% of merges.

Why 84% coverage is not evidence

Line coverage records which lines executed, not which behaviours were asserted. A test with no assertion, or one that only constructs an object, counts fully. With 900 unit tests over 220k lines, coverage is almost certainly concentrated in generated code, configuration classes, DTOs and legacy paths, while the boundary logic that actually breaks is untested. Coverage answers 'was this line reached', never 'would we notice if this logic were wrong'.

The evidence points at a specific gap

7 of 9 incidents are integration failures with eligibility and claims, whose contracts changed three times last quarter with no portal change. Browser tests cannot catch that reliably - the staging environment typically has one provider state, so a contract change either breaks the whole environment or is invisible. That gap is a missing consumer-driven contract test at each service boundary.

The gate has already failed

A check that is skipped on 38% of merges is not a control. Given a 4-hour suite with 60% fail-at-least-once, the bypass is the rational behaviour. The team has not stopped testing; it has stopped trusting the test suite, and the architecture of the suite is why.

Target shape

• Unit: thousands, seconds to a couple of minutes. Pricing, eligibility decisions, validation, formatting rules.
• Component/service integration: hundreds, minutes. Repository plus database, message handlers, HTTP clients against stubs.
• Contract tests: one suite per service boundary, consumer-driven (Pact or equivalent). Consumer CI publishes expectations; provider CI verifies them before deploy. These cover the 7-of-9 failure class and run in minutes with no browser.
• End-to-end: 50-120 tests covering a small set of revenue- and safety-critical journeys through the real stack, against ephemeral environments.
• Non-functional: performance smoke, accessibility and security scanning as separate fast checks.

Sequence - order matters more than the ratio

1. Add contract tests first. They are the fastest risk reduction, require deleting nothing, and will start failing on the real problem within days.
2. Write unit tests around boundary logic and the eligibility and claims decision paths. Use mutation testing on those packages; it tells you whether the new tests assert anything, which the coverage number never will.
3. Only then delete end-to-end tests, in reviewed batches. For each batch ask: which unit, contract or component test now covers this behaviour?
4. Re-point the surviving end-to-end suite at ephemeral environments, remove the shared staging dependency, and make it a required check under 15 minutes.

Delete explicitly: browser tests asserting form validation, field-level rules, formatting and logic already covered at unit level; duplicate journey variants that differ only in user type; tests whose sole purpose was coverage.

Judgement signal: escaped defects by category, suite duration and flake rate. A falling test count with flat or falling escapes is a win; a falling count with rising escapes means you deleted coverage rather than duplication.

Do not simply delete the label

Removing it does not restore the gate; it relocates the bypass somewhere you cannot see - force merges, admin overrides, a split change that 'doesn't need tests', or merging at 17:00 on a Friday. The label exists because it is the least-bad option available to people facing a 4-hour suite that fails 60% of the time. Treat it as a symptom with a known cause, and fix the cause while making the symptom visible.

Make skipping expensive and observable

• Require a reason, an owner and an expiry on the label; the PR cannot merge without all three.
• Publish skip rate per team weekly. Transparency with a name attached changes behaviour faster than a policy memo.
• Auto-create a follow-up ticket, and require the skipped suite to run and pass before the change can be part of a release. The skip defers cost, it does not erase it.
• Fail closed on the paths that matter: changes touching eligibility or claims integration cannot be skipped at all.
• Make the release gate, not just the merge gate, depend on the suite. Teams skip merges; they cannot skip a release they need to ship.

Fix the incentive

No gate policy will be respected while the suite is untrustworthy. Quarantine flaky tests with owners, cut the blocking suite to roughly 100 critical journeys, move the rest to a non-blocking lane, and get the required check under 15 minutes. The skip rate should fall because skipping stops being worth the trouble.

Success measure: skip rate falling while escaped defects stay flat or fall, and flake rate under 2%. If skip rate falls but escapes rise, you hardened the gate before fixing the suite - back off and fix reliability first.