Green Dashboards, Red Support Queue
A video-conferencing platform serves 12 regions. The primary dashboard shows a global availability SLI (successful joins / total join attempts) and a global latency SLI computed as p99 across all pooled requests. Both panels are green. Meanwhile support tickets from the São Paulo region have tripled over four days: users report 6-10 second waits. São Paulo carries 3% of global traffic, and 25% of joins there now exceed 6 seconds; about 2% fail after retries. The availability SLO is 99.9% globally, and the burn-rate alert fires at 14.4x over a one-hour window. Last quarter's global availability was 99.94%.
1. Why are both panels green, and what is the aggregation doing to hide this?
Read the worked answer
Run the numbers before touching the dashboard.
- São Paulo failures: 2% of 3% of traffic = 0.06% of global requests. Against a 0.1% budget, sustained for a full 28 days, that is ~60% of the window's budget. Under budget. Green.
- The burn alert needs a 1.44% global error rate to hit 14.4x. Observed is 0.06% - 24x below the alerting threshold. Never fires.
- São Paulo slow requests: 25% of 3% = 0.75% of global requests. A global p99 measures the slowest 1% of pooled requests. São Paulo's slow tail is almost entirely absorbed below that line, so the global p99 barely moves.
The aggregation is doing three separate things wrong:
- Pooling by traffic makes small regions invisible. A 3%-traffic region can be fully degraded and cost at most 3% of the global error rate.
- A single global percentile is a popularity-weighted view. It answers "is the median-ish user happy across the fleet," not "is any population of users being harmed."
- Latency is not in the SLO at all. Joins eventually succeed, so the availability SLI stays healthy while humans wait ten seconds. If your only SLI is success rate, you have declared that a slow product is a good product.
Last quarter's 99.94% is the tell: the global number was excellent while a region was suffering. That is a definition of reliability the business does not actually hold.
2. How would you restructure the SLIs and alerting so this is caught on day one instead of day four?
Read the worked answer
Keep the global SLI, add a detection layer, and add a latency SLO.
1. Keep the user-facing global SLI as the accountability number. It is the right thing to put in front of leadership, and it correctly weights by number of humans affected.
2. Add per-region SLIs as the detection layer - every region, including 3%-traffic ones. These are not all SLOs with error budgets and consequences; they are indicators with thresholds. A region at 8% error rate or p95 > 3s should page, regardless of its global contribution.
3. Add a latency SLI with its own SLO, e.g. "95% of joins complete in under 2 seconds." This is the SLO that was actually violated here, and it is invisible today.
4. Replace the single 14.4x/1h alert with multi-window, multi-burn-rate alerting:
- 14.4x over 1h (fast, pages) - page-worthy, short window
- 6x over 6h (medium, pages)
- 1x over 3d (slow, ticket) - this is, precisely, the alert that would have fired on day two of a four-day regional degradation
A one-hour-only alert is structurally blind to slow, geographically contained degradation. The slow-burn window is not a nice-to-have; it is the only alert that matches this incident's shape.
5. Weight per-region SLIs equally, not by traffic, if the business promise is "reliable everywhere." State that trade-off out loud - equal weighting means a 200-user region can page you at 3 a.m., and that is a deliberate choice you should make with the business, not accidentally.
3. Should you move to per-region SLOs with per-region error budgets? Argue both sides and decide.
Read the worked answer
The case for: a global SLO sanctions exactly this outcome. 0.06% error and a degraded São Paulo is compliant behavior under the current contract, so nobody broke a rule. If the business genuinely promises regional parity - separately sold regions, data residency, regional SLAs - then regional SLOs are the only honest representation of that promise.
The case against: 12 error budgets means 12 policy conversations, 12 freezes, and 12 opportunities for a tiny region to page a team at 3 a.m. for 40 affected users. It multiplies alerting noise and dilutes attention on the failures that matter most. It also makes error budgets fungible in a bad way: a team can look compliant in aggregate while quietly failing somewhere.
My decision - a tiered model:
- Global user-weighted SLO remains the accountability number and the basis for error budget policy and release decisions. One budget, one policy, one conversation with leadership.
- Per-region SLIs exist for every region as detection inputs with their own thresholds, feeding multi-window burn alerts. No separate error budget machinery.
- Regional SLOs with real budgets only for regions that clear a threshold - say, above 10% of traffic, or any region with a contractual SLA. São Paulo at 3% would get indicators, not a budget.
What I would not do is leave it as-is. The current setup does not fail because it is aggregated; it fails because there is no latency SLO and no slow-burn alert. Fix those first - they are cheap and they close the actual gap. The tiering decision is a follow-up written down and reviewed in a quarter.