Queueing a Six-Hour Regression Suite Behind 200 Commits a Day
Orion Insurance is decomposing a 1.2-million-line policy administration monolith into 14 services; eight are live. The regression suite holds 4,200 Selenium UI tests (6 hours 20 minutes), 900 API tests and 1,100 unit tests. It runs in full on every merge because the release manager insists nothing less is safe. The main branch takes roughly 200 commits a day across 18 teams, and the regression job queues behind a three-environment pool, so a full run typically finishes nine hours after merge. Flakiness sits at 22% on rerun. There are no consumer-driven contract tests; service teams verify integrations manually in a shared staging environment once a week. Two of the last four production incidents came from a changed field format between the quotes service and billing. Both passed the full regression suite, because the UI tests mock the billing API. Teams now want to keep the full suite as the merge gate and add more UI tests for the new services.
1. Should the full regression suite continue to gate every merge? Justify your answer with the queueing and feedback arithmetic.
Read the worked answer
The arithmetic. At 200 commits a day, a gate that completes nine hours after merge is not a gate; it is a post-hoc audit. By the time it reports, dozens of further commits have landed on top of the change, so attributing a failure is expensive and reverting is ambiguous. A gate has value only if its feedback arrives while the author still has the change in mind, which for trunk-based work means minutes.
The signal is also weak. A 22% flake rate on rerun means roughly one in five failures is false. A gate that fails one time in five for no reason is bypassed within weeks, and once bypassed its apparent safety is illusory. Fixing flakiness is a prerequisite for any gating decision, not a parallel workstream.
Recommendation. Replace the full-suite merge gate with a tiered model. Tier one, under 15 minutes: unit tests plus API tests plus contract tests for every service the change touches. Tier two, on merge to main, not blocking: the broader integration and service-level suite, with failures routed to the owning squad. Tier three, nightly and on release candidates: the full suite including the UI tests that remain. The full suite is a release control, not a merge control.
On adding UI tests. The suite already has 4,200 UI tests and still missed both incidents, because the failure mode was an integration contract, not a rendered page. More UI tests would raise runtime and flakiness while adding nothing to the class of risk that actually bit.
2. Design the test portfolio for the remainder of the service-extraction programme: what runs at which layer, and what changes as more services split out?
Read the worked answer
Anchor the portfolio on the interface between services, not on the browser. For 14 services there are at most a few dozen significant consumer-provider relationships. Consumer-driven contract tests cover each one, verified in the provider's own pipeline, and run in seconds. That is the layer that scales with the architecture; UI tests do not.
Target shape. Unit and component tests owned by each squad, covering business rules inside the service. Contract tests at every synchronous and asynchronous boundary, including message schemas. A small service-level integration suite for each service, running against real dependencies in a containerised environment. End-to-end UI tests reserved for a small number of critical journeys, roughly 50 to 150, chosen because they cross several services and represent money or regulatory exposure.
Migration path for the 4,200 UI tests. Do not delete wholesale. Cluster them by the journey they exercise, keep one per cluster, and for each discarded test ask which layer should hold that assertion, then write it there. Expect the UI suite to fall by 80% or more while coverage of risk rises.
As new services split out. Each extraction should produce contract tests before the service is cut over, plus an agreed schema versioning rule. If a squad cannot state its consumer contracts, the extraction is not ready.
Governance. Publish a portfolio dashboard by layer and by runtime, so the next request to 'add more UI tests' is answered with the current cost per assertion.
3. Two of four production incidents came from a field-format change between quotes and billing. What specific check would have caught it, and why did the existing suite not?
Read the worked answer
Why the suite missed it. The UI tests mock the billing API, so they return a canned payload shaped exactly as the test author expected. The mock is a snapshot of yesterday's contract. When billing changed the field format, the mock kept answering in the old shape and the tests kept passing. The full regression suite verified the UI against a fiction.
The check that catches it. A consumer-driven contract test: the quotes service publishes the interactions it expects, the billing service verifies those expectations in its own pipeline, and the build fails on the billing side the moment a field is renamed, retyped, or made required. This shifts detection from days-later UI execution to the provider's pull request.
Limits of that check. Contract tests catch structural drift, not semantic drift. If billing changes a currency unit from cents to dollars, or repurposes an enum value, the schema still validates and the contract still passes. Cover semantics with a small number of end-to-end tests that run against real service instances in a pre-production environment, and with explicit assertions on a handful of high-value records.
Also add. Schema validation on messages at the boundary so malformed payloads fail fast at runtime rather than corrupting downstream data, and a canary check that exercises the real integration after deploy.
Sequencing. Write contracts for the eight already-live services first, starting with quotes and billing, then make a contract verification job mandatory in every provider pipeline before further extractions proceed.