The 55-Minute PR Pipeline and a 12% Flake Rate
Northwind Retail has 220 engineers and 900 pull requests per week. Its CI pipeline runs unit tests (18 min), integration tests (22 min), end-to-end tests (15 min), and a required security scan (8 min) sequentially, giving a P50 of 55 minutes and P95 of 80 minutes. Twelve percent of runs fail for reasons unrelated to the change, mostly Selenium timing and date-dependent tests; developers rerun jobs to merge. A merge queue exists but is often bypassed near sprint end. Main has been broken on 3 of the last 10 working days. The security team requires a scan before merge for audit, but does not require it to be the last stage. Budget for additional runners is frozen for this quarter, and the CTO wants measurable improvement in 90 days.
1. Where is the highest-leverage improvement, and in what sequence would you make changes?
Read the worked answer
Start with the critical path and the flake rate, not with more runners. The pipeline is sequential, so unit tests, integration tests, and the security scan can run in parallel; that alone can remove a large block from the critical path. Shard integration and end-to-end tests by service or test file, but only after measuring stage-level duration and failure attribution. The 12% flake rate is more damaging than raw duration because reruns consume capacity and train developers to bypass the merge queue. Fix the top 10 flaky tests first; they usually account for most failures. Quarantine tests that cannot be fixed within 14 days, with a named owner and a non-blocking lane. Keep the security scan as a required merge check, run in parallel, with a fail-fast policy for critical findings. Sequence: measure, parallelize, quarantine and fix flakes, then enforce the merge queue. Do not make end-to-end tests a required PR check for every change; run them on the merge queue and nightly.
2. Design the required checks and merge queue policy. What blocks a merge, what does not, and who can bypass?
Read the worked answer
Required merge checks should be build, unit tests, security scan, and integration tests for affected services. End-to-end tests should gate the merge queue, not every PR, because they are slow and flaky. A quarantined test is visible but non-blocking, with an expiry date and owner. The merge queue should batch up to five PRs, test the merged result, and time out after 30 minutes; that catches semantic conflicts that per-PR checks miss. Bypass should be limited to an incident commander during a declared SEV-1 or SEV-2, with a reason, an audit log entry, and a follow-up ticket. Near sprint end is not a valid bypass reason. To make the policy stick, publish queue wait time and make bypass visible on the team dashboard. Also require a green main branch before new PRs enter the queue; if main is red, the queue pauses. This balances audit requirements with throughput.
3. Which metrics would you use to prove improvement to the CTO without hiding the tradeoffs?
Read the worked answer
Use a small set that reflects flow and quality. Track P50 and P95 PR pipeline duration, merge queue wait time, flake rate, rerun rate, main-branch broken days, change failure rate, and PR lead time. Establish a four-week baseline before changing the pipeline, then report weekly with confidence intervals where possible. Do not report only average duration, because a fast median can hide a long tail. Pair speed metrics with quality metrics so the team does not improve duration by weakening tests. For the security scan, report scan duration and critical findings blocked before merge. If flake rate falls but queue wait rises, the tradeoff is visible. Set targets such as flake below 2%, P95 under 30 minutes, and main broken days under one per month, and state which targets are not yet met.