Friday Deploy, Failed Rollback, and a Renamed Column
On a Friday at 16:30, ShopWave deploys checkout-service version 2.14 before a weekend sale. The new-pricing feature flag is enabled for 10% of users. Within 12 minutes, p99 latency rises from 400 ms to 2.8 s and the error rate reaches 4%. The monthly SLO is 99.9%, and the burn rate is 14x. The on-call engineer disables the flag, but latency only drops to 1.9 s. A rollback to 2.13 is attempted and fails: the release migration renamed price_cents to price_minor, and version 2.13 queries the old column. The database migration is already applied and cannot be reversed without downtime. The incident commander is on a call with support and the VP of engineering. Traffic is 3,000 requests per second, and the sale starts in 14 hours.
1. What is the immediate mitigation sequence, and what decision criteria determine when to stop mitigating?
Read the worked answer
Declare SEV-1 and assign roles: incident commander, communications lead, and database operator. First confirm the flag is fully off, including cached evaluation at the edge; if 1.9 s remains, the problem is not only pricing logic. Check database CPU, locks, and slow queries introduced by the rename. Stop further rollback attempts, because each failed rollback adds risk and noise. The fastest safe mitigation is a forward-compatible migration: add a price_cents view or generated column that maps to price_minor, then either roll back the app or deploy a patched 2.14 that uses the compatibility layer. In parallel, scale checkout-service and shift traffic away from the affected zone. Decision criteria: if p99 falls below 800 ms and errors below 1% within 20 minutes, stabilize and defer the full fix; otherwise fail over to the pre-migration database replica only if replication lag is acceptable. Communicate status page updates every 30 minutes.
2. What root causes and corrective actions should the postmortem capture?
Read the worked answer
Root causes: a schema-changing migration was coupled to an application deploy; the migration was not backward compatible, violating expand/contract; rollback was never tested against the new schema; the feature flag was enabled during a high-traffic window without canary analysis; and the deploy happened before a sale. Corrective actions: adopt expand/contract for all schema changes, with old and new code able to run during the transition. Add automated rollback tests in staging that run the previous app version against the new schema. Gate deploys with canary analysis on p99, error rate, and SLO burn, and auto-rollback on thresholds. Freeze schema changes and feature-flag enablement within 24 hours of a sale. Run a game day that practices a failed rollback. Assign owners and due dates, and track them in the same backlog as features so the actions do not decay into a document.
3. What policy changes would you recommend, and how would you enforce them rather than relying on document review?
Read the worked answer
Set a deployment policy: no schema migration and application deploy in the same change set; migrations ship first and must be backward compatible for at least one release. Require a tested rollback plan for every deploy, including database state. Define change-freeze windows before peak events, with an exception process owned by the VP of engineering. Limit feature-flag enablement to business hours unless the flag has a tested kill switch and an owner on call. Add a deployment checklist that includes canary duration, SLO burn threshold, and communications plan. Finally, make rollback drills quarterly and publish the results. Enforce with CI checks where possible, such as blocking a migration without a compatibility annotation or blocking production deploys during a declared freeze. Policies enforced by tooling survive personnel changes better than review documents.