Canary at 5% and the Error Budget Is Already Gone

Defects.

1. The threshold permits a regression. An absolute 1.0% gate against a 0.35% baseline allows a ~2.9× degradation to pass, and 0.9% was a 2.6× regression. The gate should be relative to the current baseline, not a fixed number.
2. Wrong signals. 5xx and p95 miss lock contention, which shows up in p99, in database lock waits, and in business completion rate. p95 hides a slow tail affecting a minority of checkouts - the most expensive kind.
3. Fixed window ignores traffic. At 200 rps, 5% = 10 rps, so 10 minutes = 6,000 requests; at 0.35% that is ~21 expected errors (sd ≈ √21 ≈ 5), so a jump to 1.4% (84 errors) is comfortably detectable. But at 02:00 traffic may be 40 rps: 5% = 2 rps → 1,200 requests → ~4 expected errors, and you cannot separate 0.35% from 1.4%. The window must be sized by request count, not by clock.
4. App and schema shipped as one unit, so "rollback" restored pods but not the system.

Corrected configuration. Success criteria per step: 5xx rate ≤ max(baseline + 0.1pp, baseline × 1.3); p99 latency ≤ 1.2 × baseline p99; checkout completion rate within 2% of baseline; database lock-wait time and replication lag within baseline. Require at least 10,000 requests per step, extending the pause automatically when traffic is low. Compare against a rolling baseline computed from the previous 24 hours at the same hour of day, and use a statistical test (e.g., two-proportion z-test at 99%) rather than a point threshold. Add a business-metric gate so a silent functional regression fails the rollout even when infrastructure metrics look healthy.

Use expand/contract and treat schema change as a separate deployable unit from application change.

Expand (deploy first, alone). Add the new column, table or index in a way that is invisible to the running code and safe under load: CREATE INDEX CONCURRENTLY, ALTER TABLE ... ADD COLUMN nullable with no default rewrite, backfill in batched jobs off-peak with a lock_timeout set so it fails fast instead of queuing behind a lock. Verify with a plan and a dry run on a production-sized copy. The old code must run perfectly against the expanded schema; nothing reads the new structure yet.

Migrate (code deploy). Ship application code that writes both representations if needed and reads through a feature flag, so flipping the flag is the actual release and rollback is a flag change taking seconds.

Contract (later, separate release). Drop the old column or index only after the new code has been stable for at least one full release cycle and no rollback to old code is expected.

Under this sequencing, rollback means rolling back pods only - the schema is compatible in both directions, so the previous version still works against the expanded database, and the index rebuild has already finished safely before any code change. The failure here was that a lock-taking DDL ran as part of the application release, so rollback could not restore the system's performance. Add a pipeline rule: a release containing DDL requires the DDL to have been applied and verified in a prior step, and the rollout must fail closed if that step did not run.

Arithmetic first. A 99.9% monthly SLO gives 43.2 minutes of budget. 80% consumed leaves roughly 8.6 minutes - about 0.02% of the month - for the remaining ~20 days. Any further incident of even moderate size breaches the SLO, and a breach means the SLO review and the customer credits that go with it.

This week. Do not ship the Friday 17:00 release. Freeze feature deploys to production; permit reliability fixes, and permit security patches with a named approver and a tightened canary (1% step, minimum 20,000 requests, 30-minute dwell, automatic abort). Move the release to a Tuesday or Wednesday morning with the full on-call rotation staffed and a verified, tested rollback path for every component including schema. If the release contains a migration, it must go out as a separate expand step at least a day earlier.

Policy. Encode the budget state as a gate, not a discussion:

• Budget > 50% remaining: normal release cadence.
• 25–50%: canary required, no Friday deploys, release review by a second engineer.
• < 25%: feature freeze; only reliability and security changes, each with an explicit risk note and an approver.
• Exhausted: full freeze until the SLO review, with an exception process that a director must sign.

The point is that the decision is made once, in the policy, rather than re-argued every week under shipping pressure. Also review whether the third-party outage should be excluded from the SLO or tracked under a separate dependency SLO - but do not retroactively edit this month's budget to make the numbers look better.