Blue/Green With a Shared Database and a Spent Error Budget

The flaw is the shared stateful dependency. Blue/green isolates compute. It gives you two application versions and a traffic switch. It does not give you two databases here - both colours point at the same PostgreSQL instance, so the schema is a single shared resource. The instant the migration runs, it applies to blue and green simultaneously. Rolling back the application image does not roll back the schema.

Two immediate hazards in the migration itself:

• Adding a NOT NULL column without a default fails outright on a table with existing rows, or takes a heavy lock while rewriting 400 million rows, which stalls writes during the very window you are switching traffic.
• Dropping region_code means any still-running old version crashes on read. In a shared-database blue/green, the old colour is your rollback plan - and this release destroys it.

Sequence it as expand/contract, one deploy per step:

1. Expand: add tenant_id as nullable, with a safe default for new writes. Deploy code that writes both columns and reads the old one. Backfill in batches with throttling; monitor replication lag and lock waits.
2. Migrate reads: deploy code that reads tenant_id with a fallback to the backfill path. Add the NOT NULL constraint using a validated-constraint approach rather than a full rewrite.
3. Contract: only after telemetry shows zero reads of region_code for a full release cycle, drop it in a migration that is not coupled to an application release.

Each step is backward and forward compatible, which is the actual precondition for blue/green to mean anything.

Do the arithmetic first. 0.4 of 0.5 percent consumed leaves 20 percent of the monthly budget - roughly 0.1 percent of requests, over the remaining days of the window. That is thin. An 11-minute rollback on a high-traffic service could consume it entirely.

Decision: do not ship R-42 as specified this week. Ship the expand step only, because it is additive, backward compatible, and reversible without a table copy. Hold the constraint and the drop.

Reasoning to show the examiner:

• A contractual date is a business constraint, not a technical one. It changes who accepts the risk; it does not change the rollback time. If the feature genuinely needs the column, an additive, dual-write release behind a feature flag delivers it without the destructive part.
• The error budget exists precisely to make this a rule rather than an argument. Having spent 80 percent of it, the policy should default to freezing risky changes; overriding it requires an exception with named owners.

What would change my answer:

• Migration is proven expand-only, runs in seconds, and the old column drop is deferred to its own change.
• A canary at 1 percent carries the new build with automated rollback on SLO burn rate, and shows no error-rate or latency regression over a meaningful soak.
• A rehearsed rollback in staging measures under two minutes, not eleven.

What would not change it: a verbal assurance that the migration is tested, or a promise to watch the dashboards.

Rollback is a schema property, not a deployment property. In a shared-database blue/green, rollback means flipping traffic back to the previous colour - which only works if the previous colour can still run against the current schema. That is why expand/contract is the plan, not an optimisation.

The plan:

1. Precondition check before any switch: the migration is additive; the old build reads and writes successfully against the new schema (verified in staging). If this is false, there is no rollback and the release should not proceed.
2. Pre-warm green with synthetic traffic and a health gate on real dependency checks, not just a TCP port check.
3. Switch a slice, not everything: 1 percent, 10 percent, then full, with automated evaluation of error rate, p95 latency, and DB connection counts at each gate.
4. Rollback = flip the routing rule back, then keep both colours running until the canary signals are clean - do not tear down blue during the watch window.
5. Feature flag kill switch for the new behaviour, independent of the deployment, so a logic bug does not require any traffic movement.

What actually makes it fast:

• Migrations are forward-only and additive - never a down-migration that copies a table. That single rule would have turned the previous 11 minutes into seconds.
• Artifacts are retained and immutable: the previous two image digests are pinned by tag, so rolling back does not involve rebuilding.
• The rollback is rehearsed and timed in a game day, because the number that matters is the measured one.
• The traffic switch is a routing change, not a DNS change with a long TTL.

Verification: after rollback, confirm error rate returns to baseline, connection pool usage drops, and no partial writes landed from the new schema path. Record the measured rollback time in the release notes so the next error-budget conversation is factual.