Canary Release on a Payments Schema Change

Volume and duration

• 5% of 900 rps is 45 rps; ten minutes is about 27,000 requests. That is enough to see large breakage and not much else. A regression that raises declines from 0.4% to 0.6% produces around 54 extra declines in the window, which will not cross a 1% threshold.

The threshold measures the wrong thing

• A declined authorization is an HTTP 200 with a business failure in the body. It is not counted as an error. A doubling of declines can happen with a 0.00% error rate, so the rollback condition will never fire for the failure mode that matters most.
• A 1% ceiling also tolerates roughly 2.5x the normal decline rate before acting.

The canary cannot fix the irreversible part

• If the column is NOT NULL and the old code path does not populate it, rolling back the code leaves the database in a state the old code cannot write to. The canary protects the code; it does nothing for the schema.

Other gaps

• Ten minutes does not cover a settlement or batch cycle, cache expiry, or downstream reconciliation.
• Retries during a bad canary amplify load and can create duplicate authorization attempts.
• There is no comparison against a same-hour baseline, only a static threshold.

Use expand/contract: never combine a schema change and a code change in a way that makes either irreversible.

1. Expand. Add the column as nullable with no application change. Gate: deploy is a metadata-only change; verify it applied without a long lock.
2. Dual-write. Deploy code that writes both the old and new paths and still reads the old one. Reverting this is a normal code rollback, seconds to minutes. Gate: 24 hours with zero dual-write errors and matching row counts.
3. Backfill. Run a throttled, idempotent, resumable job over 400 million rows in batches, monitoring replication lag. Gate: checksums and counts match; lag stayed within tolerance.
4. Shift reads. Move the read path to the new column behind a feature flag and ramp 1% to 100%. Gate: decline rate and latency parity against the same-hour baseline, not against a fixed threshold.
5. Constrain. Validate the NOT NULL constraint after the old write path is gone, then drop the old path in a later, separate release.

The key property: at every step the previous state is still valid, so a rollback never has to undo a data migration. The error budget is protected because the risky step is small and reversible.

Decision: stop and roll back the routing now.

• 0.9% versus 0.4% is more than a doubling of the highest-risk failure mode. That is not a threshold question; it is a customer-impact question.
• Retries compound it: every decline that is retried multiplies authorization attempts, adds load to the very path that is degrading, and can produce duplicate captures if retries are not idempotent.
• Error budget is 12 of 43 minutes spent. Spending more on an ambiguous signal is not a defensible trade.

Rollback action

Route traffic back to the previous ReplicaSet, which should still be warm, so the switch takes seconds. Leave the expanded column in place - it is additive and harmless. Do not attempt to reverse any schema step.

What to gather before retrying

• Is the delta statistically real, measured against the same hour on prior days rather than against a static threshold?
• Is it concentrated in one issuer, card network, BIN range, geography, or downstream dependency? Concentration points to an integration problem, not a code-wide defect.
• Are retries idempotent, and how many duplicate authorizations were created?
• Does the new write path change latency to the processor, which could push requests past an authorization timeout?

Only after those answers should the release be re-attempted, and then with a decline-rate-based gate rather than an error-rate gate.