Blue-Green Cannot Undo a Dropped Column

No. Blue-green solves the application half and creates false confidence about the rest.

• Blue-green gives two identical stacks, lets you warm and smoke-test green, and cuts traffic over at the load balancer so a bad application build reverts in seconds. That is real value.
• But both colours share one database. The destructive migration is the actual failure, and no cutover instant changes it: once the legacy column is dropped, the blue version is dead too. The rollback was impossible before the deployment method was chosen.
• What the incident requires is expand/contract. Release 1 adds the new nullable column and dual-writes to both. Backfill in batches. Release 2 reads the new column and stops writing the old one. Release 3 drops the old column after the rollback window closes - say two weeks of clean operation.
• The governing rule: every schema change must be backward compatible with the currently running application version, because during any real deployment two versions coexist.
• Also note blue-green roughly doubles infrastructure cost, needs sticky or externalized session and cache state, and does not remove the 20-minute window unless combined with rolling or canary traffic shifting.

The design misses the RPO by roughly two orders of magnitude.

• Nightly fulls give an RPO of up to 24 hours against a 15-minute contractual requirement. More frequent full dumps are the wrong instrument: they are expensive and still coarse.
• Move to point-in-time recovery. PostgreSQL: a weekly base backup plus continuous WAL archiving with archive_timeout around 60 seconds to force segment switches during quiet periods. MySQL: binlog streaming with a one-minute flush. That yields an RPO measured in minutes.
• RTO must be measured, not assumed. Restoring tens of gigabytes plus replaying WAL, then reconfiguring DNS and verifying, is the number to record, and it must fit inside four hours with margin.
• An untested backup is a hypothesis. Run a quarterly restore drill into an isolated environment and validate with row counts and checksums against production.
• Keep a copy in a separate account or region with object-lock immutability, so a stolen credential cannot delete the recovery path along with production.
• Write the restore runbook down and time it; the incident is not the moment to discover who holds the encryption key.

Decide the abort conditions before the deployment, not during the outage.

Gate: shift 5% of traffic to the new version, hold 15 minutes, then 25%, 50%, 100%, with a decision at each step.

Automatic rollback if any of the following holds on the new version:

• error rate exceeds the control version by more than 1 percentage point absolute over a 5-minute window;
• p95 latency exceeds 1.5× the baseline for the same endpoint;
• any SLO error-budget burn rate above 10× over one hour;
• a canary-only alert fires.

Two disciplines make this work. Measure per version, never in aggregate - blended metrics hide a broken canary at 5% traffic. And compare canary against control simultaneously so ordinary traffic drift does not trigger or mask the gate.

The misleading metric is HTTP 200 rate on the web tier: a service can answer 200 while checkout fails because a downstream call times out. Use a synthetic business transaction (login, quote, create shipment) as the primary signal, and require that rollback is one command that actually works - which it only does if the schema change was backward compatible.