Trellis Media: an SLO conversation that is really a release conversation

Reject it - not because ambition is bad, but because the arithmetic makes 99.99% a freeze order, and a budget nobody can meet gets ignored rather than respected.

The arithmetic: a 28-day window is 40,320 minutes. 99.9% allows ~40 minutes. 99.99% allows ~4 minutes. At the current burn rate of 34 minutes in the last window, one incident of the size you already had once this month is more than five times the entire new budget.

Now multiply by change rate. 40 deploys a week is about 170 a month. Observed MTTR on the two incidents was roughly 12-22 minutes. To hold 99.99% you need either a near-zero defect rate on deploys or an MTTR under about two minutes. Neither follows from changing a number in a dashboard. What actually follows is that the team starts classifying incidents as 'not really downtime' to keep the number green, and the SLO stops measuring anything - which is worse than not having one.

Also challenge the SLI before the target. Availability at the load balancer alone will not capture the memory leak properly if its main symptom was latency, not errors. A 12-minute incident where p99 goes from 180 ms to 4 s may barely dent an availability SLO. Define a second SLI - proportion of requests served under a latency threshold - and give it its own budget. Otherwise you have a target that the team can hit while users are unhappy.

What I would say instead: keep 99.9% as the external commitment, add the latency SLI, and set an internal target of 99.95% with a gate: when the error budget is exhausted, releases pause until it recovers. That is a real forcing function and it is achievable. Revisit 99.99% after two quarters of burn under 20% of budget.

Attack the 22-minute class, because it is 65% of the burn and it is the one you control end to end.

Breakdown of the 34 minutes:

• Bad configuration push: 22 min (65%).
• Memory leak: 12 min (35%).
• Zone failure: 0 min - a win, and evidence worth writing down.

The natural engineer instinct is to fix the memory leak, because it is a code defect with a satisfying root cause. But it is a single bug with a long tail of similar bugs behind it, and each one costs 12 minutes when it hits. The configuration push is a class of change: any config change, any deploy, 40 times a week, with a mechanism that has no blast-radius limit and 22-minute detection. Fixing the mechanism retires the entire class.

Concretely, in order of leverage:

1. Progressive rollout with automatic rollback. Ship to one backend group with 1% of traffic, hold, then 10%, 50%, 100% with an automated gate on error rate and p99 versus the previous version. The 22 minutes was human detection plus human rollback; automation makes it under 3.
2. Config validation in CI - schema checks and a dry-run render against the staging backend. Config changes are the highest-frequency change type with the least testing, which is why they produce incidents like this one.
3. Alert on SLO burn rate, not on static thresholds. A fast-burn alert (1-hour window consuming 14x the sustainable rate) pages in minutes instead of waiting for a dashboard to look red. The 22-minute duration is a detection problem before it is a rollback problem.
4. Then the memory leak, with a heap-usage alert and a memory limit so the failure is a restart rather than a degradation.

Why not a change freeze: 40 deploys a week is what lets the team move. The fix is to make deploys smaller and reversible, not rarer - and the error-budget policy (pause on exhaustion) is where the freeze belongs, triggered by data rather than by anxiety.

No. The evidence points away from it, and it would consume the entire reliability budget for a failure mode that has not occurred.

Evidence:

• The only infrastructure failure in the window - a zone failure in us-east1 - cost zero user-visible minutes because draining worked. That is exactly the scenario multi-region is bought for, and you already handle it.
• 100% of the user-visible burn came from change (config push) and application-level defects (memory leak), neither of which active-active fixes. A bad config pushed to three regions is a three-region outage, and it is harder to roll back.
• Both backends already sit behind a global load balancer, so region-level failover is a routing decision, not a rewrite.

What active-active would actually cost a 9-person team: dual-write or globally consistent data design for the metadata store, conflict resolution, split-brain and asymmetric-partition runbooks, roughly 3x the on-call surface, a training burden, and a new class of subtle data bug. Nine engineers with no SRE title and 40 deploys a week do not have the slack to absorb that without dropping the release engineering work that would actually help.

Instead:

1. Prove the failover you have. Run a GameDay: drain europe-west1 entirely and measure user-visible errors. If it is clean, you have evidence to show leadership and a runbook that has been executed once. If it is not clean, fix that - it is a fraction of the cost of a rewrite.
2. Capacity headroom: verify each region can absorb 100% of traffic at p99 within budget on a failure, and set the load balancer's failover policy and health checks accordingly. N+1 capacity is cheap; a rewrite is not.
3. Put a price on the ask. Quantify cost and engineering-months, and set the trigger explicitly: 'we revisit active-active if a full-region outage costs more than 50% of a quarter's budget, or if a contractual SLA moves to 99.99%.' Leadership asked for the rewrite because they wanted a reliability guarantee. Show them the guarantee you can give them this quarter, and the data that justifies it.