Eleven Days Between Cause and Discovery
Corvus Analytics runs a B2B SaaS platform on Kubernetes for 900 enterprise customers. Developers merge into 'develop'; a nightly job merges develop into 'release'; QA deploys release to staging every two weeks. A release manager cuts a release branch, and a four-day manual regression pass precedes production. A typical release carries 37 commits from nine developers. Integration failures are discovered on average 11 days after the commit that caused them. 18% of automated tests fail intermittently, and the team's standard practice is to rerun until green. Three of the last five production incidents were traced to configuration or database schema changes that had passed staging untouched, and the DBA applies migrations by hand during the release window. A hotfix branch has been ahead of the mainline for six weeks. Leadership has committed to daily production deployment within six months.
1. Which constraint do you attack first, and why? Justify the ordering using lead-time and feedback arguments.
Read the worked answer
Attack the 11-day feedback delay first - not deployment frequency. Daily deployment of a pipeline that detects breakage 11 days late simply ships breakage daily, faster.
Reasoning by constraint:
- Lead time is dominated by queues, not work. Nightly merge (about a day), biweekly staging, four-day regression, release window. Little's Law: 37 commits per release with a two-week cadence means large batches, and a defect's age at detection scales with batch size. Shrinking batch size attacks detection time and defect-finding cost simultaneously.
- Second: hand-applied migrations and config. Those are precisely the change types escaping staging, which means the pipeline is not the path to production - the DBA and the release window are. Daily deployment is impossible while a human gate owns the riskiest class of change.
- Third: the six-week hotfix divergence. It is a symptom of slow, risky releases. Merge it now and it will re-diverge next cycle unless mainline becomes releasable.
First concrete move: trunk-based development with branches living under a day, continuous merge to main, and a pipeline that can take any green main build to production behind a flag. Track commit-to-production lead time and, specifically, time from defect introduction to detection - currently 11 days and the number to drive down first.
2. Describe the target branching, pipeline and database change model, including how configuration and schema reach production safely.
Read the worked answer
Branching: one mainline, short-lived feature branches, releases as tags rather than branches. Feature flags decouple deploy from release. Hotfixes are ordinary commits to main through the same pipeline.
Pipeline stages, each fast and gating: build and unit tests (under 10 minutes) → contract tests → deploy to an ephemeral preview environment per change → automated acceptance on the preview → production canary at 1-5% of traffic for 10-30 minutes with automated rollback on SLO breach → full rollout. The shared long-lived staging environment is either retired or demoted to a smoke environment; the preview environment becomes the pre-production reality.
Database changes: expand/contract. Migration N adds the new column or table and backfills; code deploys reading both shapes; migration N+1 removes the old path once no code references it. Migrations are versioned, idempotent, run automatically by the pipeline before the application deploy, validated in CI against a production-like schema, and must be backward compatible with the running version so rollback is possible. Never pair a destructive migration with the release that stops using the old shape.
Configuration: config as code, versioned in the same repository, deployed through the same pipeline, with environment-specific values in a config service that keeps an audit trail. Config changes get the same canary and rollback treatment as code - this alone would have caught the three incidents that passed staging untouched.
Rollback: every deploy revertible in under five minutes, preferably by flag flip. Make that a pipeline exit criterion, not an aspiration.
3. How do you reduce flaky tests and retire the four-day manual regression without relying on a testing hero, and what do you measure?
Read the worked answer
Flaky tests: treat them as a defect backlog with an owner and a budget.
- Quarantine automatically. Any test that fails and then passes on rerun without a code change moves out of the blocking suite into a tagged quarantine suite, and must be fixed or deleted within 10 working days. Rerun-until-green is banned in the blocking suite.
- Root-cause the top patterns - time, ordering, shared state, network, async waits. Flakiness clusters; a handful of fixes usually removes most of the 18%.
- Deleting low-value tests is a legitimate outcome. A small trustworthy suite beats a large unreliable one, and trust is the thing being lost right now.
Manual regression: replace it slice by slice, prioritised by incident history rather than test-plan completeness.
For each of the four days, list what it actually catches. Automate the highest-risk items; for the remainder, make an explicit written decision to accept the risk with a mitigation (canary plus flag). Keep exploratory testing, time-boxed, as a complement to automation rather than as a release gate.
Measure: blocking-suite flaky rate (target under 2%), blocking-suite duration, escaped defects per release, percentage of regression checks automated, canary rollback rate, and time from defect introduction to detection. Two consecutive releases where the manual pass found nothing the automation missed is the evidence to cut the pass to one day, then to canary plus smoke - not a date on a plan.