Canary Releases, a Locked Table, and a Failed Rollback
ShopDirect deploys its e-commerce platform about 15 times per week using canary releases and feature flags. The platform includes a Java monolith, four Node.js services, PostgreSQL, and Redis. Twice this month a release passed all automated tests but had to be rolled back because a database migration locked a table for 11 minutes and the rollback script failed on a foreign-key constraint. Test environments are refreshed weekly from production but masked; the staging database is 10% of production size. Feature flags are set per environment, and one incident involved a flag enabled in production but not in staging. Mean time to restore is 45 minutes against a 30-minute target. There are two DBAs, no dedicated SRE, and a change advisory board still reviews high-risk changes. The team wants to move to continuous deployment with automated quality gates.
1. Design the quality gates and test stages for the pipeline so that database migration and feature-flag risks are covered. State what runs at commit, pre-deploy, canary, and post-deploy.
Read the worked answer
Commit stage: fast unit tests, static analysis, contract tests between the monolith and Node.js services, and a small API smoke set. Gate on build and unit/contract success; keep it short.
Pre-deploy: run migrations against a production-like clone, not the 10%-size staging database. Check that the migration is backward compatible, measure lock duration against an agreed threshold, and execute the rollback script on the clone. Add a flag-parity check comparing flag configuration between staging and production. Gate if the migration locks beyond threshold, rollback fails, or flags differ without an approved reason.
Canary: route a small slice of traffic, then automatically compare error rate, latency, checkout conversion, and database wait events against the baseline. Roll back automatically on breach.
Post-deploy: synthetic checkout journeys, data-integrity checks, and flag-state verification. Because rollback itself failed before, test the rollback path as part of the gate, not after an incident. The 30-minute MTTR target means these checks and rollback actions must be automated.
2. Recommend a database migration and release strategy that reduces rollback risk. Compare canary and blue-green for this situation, and explain how feature flags fit.
Read the worked answer
The core problem is that schema changes are not instantly reversible. Use expand-and-contract migrations: add nullable columns or new tables, deploy code that writes both old and new shapes, backfill, switch reads, and only later remove old structures. Never combine a destructive change with the first release that uses it.
Canary versus blue-green: canary reduces application risk by exposing a small traffic slice, but both versions share the database, so it does not make an incompatible schema safe. Blue-green gives fast application switching but has the same shared-database problem and doubles environment cost. Neither substitutes for backward-compatible migrations. Use canary for the application, and validate migrations on a production-like clone with realistic data volume and lock timing.
Feature flags separate deploy from release: deploy dark, enable for internal users, then progressively enable. They only help if flag configuration is tested and promoted consistently; the staging mismatch incident shows why parity checks and flag audits are needed. Prefer forward fixes over rollback scripts where possible, but still test rollback for high-risk changes.
3. How should ShopDirect reduce mean time to restore and adapt the change advisory board without losing control of high-risk changes?
Read the worked answer
Reduce detection and decision time rather than adding approvals. Establish service-level dashboards for errors, latency, database locks and wait events, and checkout conversion; define automatic rollback or flag-disable triggers. Write runbooks for the known failure modes, including failed migration rollback. Practise them in game days, because the previous rollback script had never been exercised against a foreign-key constraint.
For the CAB, move to risk-based change classification. Low-risk, reversible changes with automated gates and flag control can proceed without board review. High-risk changes, especially destructive schema changes, new payment paths, or flag changes affecting all users, still require review with DBAs and business owners. Use error budgets and post-incident reviews to adjust classification. Track MTTR, change failure rate, and the percentage of rollbacks that succeeded, aiming to bring MTTR below 30 minutes.