Blue/Green Deploy with a Schema Migration That Cannot Roll Back
A payments team runs a containerized API on ECS Fargate behind an ALB with two target groups. A CodePipeline deploys a new task definition, shifts 10% of traffic for 5 minutes, then 100%. The release adds a NOT NULL currency_code column to a 40M-row transactions table and backfills it in the same migration. The previous task definition writes rows without currency_code. During the 10% step, error rate stays flat, so the pipeline promotes to 100%. Twenty minutes later, checkout failures spike: the old tasks still draining connections write NULL values that violate the constraint, and a rollback to the previous image now fails because the column is NOT NULL. P99 latency on the new version is 180ms vs 120ms baseline.
1. The canary step passed but the release still caused a production incident. What does this reveal about the pipeline's promotion criteria, and how should the deploy be restructured?
Read the worked answer
The canary gate only measured the new version's error rate over 5 minutes. It never exercised the old version writing against the new schema, which is the actual compatibility boundary. Two changes:
- Make the migration expand/contract and decouple it from the deploy. Ship schema changes as separate, backward-compatible steps: (a) add
currency_codeas nullable with a default, (b) deploy code that writes it, (c) backfill in batches, (d) only then add NOT NULL in a later release. A rollback of app code is then always safe because the schema accepts both writers. - Gate on cross-version behavior, not just the canary's own metrics. During the canary, keep old and new tasks serving simultaneously and assert on end-to-end business metrics (checkout success), not just HTTP 5xx. Add an automated check that the previous task definition can still write to the migrated schema before promoting.
The 180ms vs 120ms P99 gap is a separate signal: a 50% regression should have been a promotion blocker even with flat error rates. Gate on latency budget, not error rate alone.
2. The team proposes adding an automatic rollback trigger on ALB 5xx. Explain why this would not have saved them and what the rollback trigger should key on.
Read the worked answer
The failure mode was constraint violations from the old tasks during connection draining, which surface as 5xx on the old target group, not the new one, and only after full promotion. An ALB 5xx alarm scoped to the new target group would have stayed green. Even a service-wide 5xx alarm would fire after the damage and roll back into a schema that no longer accepts the old writer, so the rollback itself fails.
Rollback triggers should key on:
- Business-level success rate (checkout completions) aggregated across all target groups, not per-version HTTP codes.
- Deploy-correlated change detection (CloudWatch anomaly alarms on the composite metric) with a short evaluation window during and after promotion.
- A precondition check that the rollback target is actually deployable against current schema state; if not, the trigger should halt and page rather than attempt a bad rollback.
The lesson: an automatic rollback is only as good as the assumption that the previous version still works. Schema-compatibility checks belong in the pipeline, before promotion.