A Canary That Could Not Roll Back a Dropped Column

Two distinct failures

1. The canary monitored the wrong property. A 3%/10-minute canary asks 'is the new version healthy?' The failure here was 'can the old version still work against the new schema?' The old pods were not throwing new errors, adding latency or failing readiness; they were doing exactly what they were designed to do against a column that no longer existed. A 5xx threshold measured on new-version traffic can look clean while the deterministic breakage sits in the old version.
2. The change was irreversible. Rollback reverts code. It does not restore a dropped column, deleted rows or narrowed types. The automatic rollback path was structurally incapable of helping, and the 25-second grace period meant old code was still executing well into the incident window.

The class rollback cannot cover

• Destructive or non-backward-compatible DDL: drop, rename, narrowing a type, adding NOT NULL without a default.
• Data deletion or in-place transformation that overwrites values.
• External side effects already emitted: payments captured, emails sent, webhooks fired, events published to a queue.
• Messages already consumed by the new version from a queue.
• Any situation where old and new versions cannot both run.

The transferable rule

Reversibility is a property of the change, not of the deployment tool. A canary gate is only meaningful for changes that are backward compatible with the version currently serving traffic. Before approving a deploy, ask what the rollback actually restores - if the answer is 'the code, but not the state', the canary is decoration and the change needs the expand/contract treatment instead.

Step 0 - inventory the window, not just the code

Find every reader and writer: application paths, ORM mappings, reporting queries, analytics replicas, batch jobs. The relevant safety window is not the 25-second pod lifetime; it is the longest-lived thing that touches the column. In a billing system that is almost certainly the monthly invoice run.

Step 1 - expand (additive only)

If a replacement column or table is needed, add it and backfill, with dual-write if both are live. Nothing reads the new path yet. Authorising check: migration applies without a long lock, and dual-write is verified by comparing counts.

Step 2 - stop reading

Deploy code that no longer selects legacy_ref. Authorising check: query telemetry (pg_stat_statements or slow-query logs) shows zero statements referencing the column for a window longer than the maximum instance lifetime plus one full business cycle - for billing, a month, or a deliberately forced run of the monthly job. Also confirm no BI or warehouse connection references it.

Step 3 - stop writing

Deploy code that no longer writes it. Often the same deploy as step 2; if separate, the same verification applies.

Step 4 - contract, separately

Take a snapshot or verified backup, announce a freeze, then drop the column in its own deploy with no application change attached, so that if it fails the recovery path is trivial.

Why the sequence matters

The incident collapsed steps 2-4 into one deploy, converting an ordinary schema change into a one-way door. Each step here is independently deployable and reversible up to step 4.

The signal is that state, not compute, is the constraint

Blue-green isolates compute. If both colours share a database, a queue and a scheduler, they are not isolated - they are two concurrent writers to the same state. Green picking up blue's queued jobs proves it: the queue is a version-less channel, so work crosses the boundary regardless of colour.

Two questions drive the pattern choice

1. What is shared and mutable? Database, queue, cache, object storage, outbound email, scheduled jobs, rate-limited third-party APIs.
2. Can two versions run simultaneously and both be correct? If yes, canary or rolling is the right default. If no, either make them compatible or fence the shared resource.

What to do for Arcadia

• Default to canary/rolling plus an N-1 compatibility obligation: the previous version must keep working against the new schema and configuration for at least the maximum rollout window. Test that in CI by running the old image against the new schema - this is the check that would have caught the incident.
• Version the queue: a queue per version, or backward-compatible consumers with schema-versioned messages. For scheduled jobs, use leader election or a distributed lock so exactly one version runs them during a transition.
• Use feature flags to separate deploy from release: expose 1% of billing traffic to the new engine with a flag, and turn it off instantly without depending on infrastructure state.

When blue-green is still right

Stateless services, or cutovers where you can also switch the database connection and stop the old colour's writers before green starts. The principle: choose the pattern from the compatibility of shared state, not from the appeal of instant rollback.