Checkout API: the SLI that says everything is fine while users fail
Your team owns a checkout API serving about 40,000 requests per minute across three regions. The availability SLO is 99.95% over a rolling 28 days, measured from Envoy sidecar metrics as the ratio of non-5xx responses to total responses. Last quarter the sidecar SLI reported 99.97% and the team shipped features on schedule. Meanwhile, support tickets about failed checkouts tripled, and the mobile SDK's own telemetry shows a 4.2% failure rate in APAC versus 0.3% in the US. Two details surfaced in the review. First, the mobile client retries failed checkout POSTs up to three times. Second, the CDN in front of the API returns a cached "please try again" page with HTTP 200 when the origin times out. Finance is asking why an error budget that looks healthy is producing customer complaints.
1. Is the sidecar-based SLI a valid measure of user-perceived availability for this SLO? Justify your answer and describe the SLI you would adopt instead.
Read the worked answer
No. It is biased upward in three independent ways, and each one is a measurement-boundary error rather than a bug.
- Requests that never reach Envoy are invisible. APAC mobile traffic that fails at DNS, TLS, or carrier network, and requests that time out at the CDN before the origin responds, never enter the numerator or the denominator. A failure the user saw contributes nothing to the SLI.
- Retries dilute the ratio. One user action becomes up to four requests. A checkout that fails twice and succeeds on the third attempt contributes three successes and one failure to a per-request ratio, so it scores 75% instead of 0%.
- The CDN launders failures into 200s. A cached error page returned with HTTP 200 is either counted as a good request or not sampled at all by Envoy metrics.
An SLI is a ratio of good events to valid events, and both must be counted at the boundary the user actually experiences. Adopt a checkout-attempt SLI: valid event = a distinct checkout_id submitted; good event = that checkout_id reaches a terminal success state within 10 seconds. Derive it from edge access logs joined to a server-side checkout_completed event, deduplicated by checkout_id so retries collapse to one event. Cross-check against client RUM for the APAC gap. Separately, change the CDN to return 504 rather than a 200 error page. Keep the sidecar SLI for diagnosis, but not as the SLO.
2. The 28-day budget for 99.95% is small. Compute it, estimate what the observed APAC failure rate consumes, and state the policy action you would take given the team's error-budget policy.
Read the worked answer
Budget arithmetic. 28 days = 40,320 minutes. A 99.95% objective allows 0.05% of that as unavailability: 40,320 × 0.0005 = 20.16 minutes of full outage, or equivalently 0.05% of events.
What APAC consumes. If APAC is roughly 20% of 40,000 rpm, that is 8,000 requests/minute at a 4.2% failure rate, or about 336 failures/minute. Over the window that is 336 × 40,320 ≈ 13.5M failed attempts against a budget of 0.05% of ~1.61B total requests ≈ 806k. The observed rate burns roughly 17× the budget. Even at a more conservative 10% APAC traffic share it is still about 8× over.
Policy action, with one guardrail first. The honest conclusion is that the SLO is being blown and the error-budget policy (freeze feature work, redirect the team to reliability) should fire. But the new SLI has never been validated, and burning budget retroactively against an unproven metric invites gaming and argument. So: run the checkout-attempt SLI and the sidecar SLI in parallel for two weeks, confirm the new number tracks support tickets and RUM, then declare the SLO breached and invoke the policy. In parallel, open a specific remediation item for the APAC network path and CDN timeout behaviour, since those are the two mechanisms the data already implicates.
3. Product now wants a latency SLO for checkout as well. What would you scope, and what pitfalls would you flag?
Read the worked answer
Scope it as a threshold SLI, not an average. Define latency as the proportion of checkout attempts completing in under a chosen threshold, measured end-to-end from the client's first byte sent to the checkout_completed event. Pick the threshold from the observed distribution at a meaningful percentile (for example the current p95), not from a round number, and pick a threshold that correlates with abandonment - a fast-but-failed checkout is not a win, so latency should be evaluated on attempts that succeeded and abandonment measured separately.
Pitfalls to flag:
- Retries destroy latency semantics. If the client retries, the user-perceived duration includes the failed attempt plus backoff. Use the user-action span, not the server request, or the metric will look far better than reality.
- Server-side latency excludes the part users complain about. CDN and mobile network time can dominate in APAC. Measuring at Envoy recreates the same measurement-boundary mistake as the availability SLI.
- Averages hide the tail. Use a distribution - histogram buckets or native histograms - so you can compute multiple thresholds later without re-instrumenting.
- Two SLOs interact. A latency SLO that excludes errors can be met by fast-failing. State explicitly that error responses count as bad events for both objectives, or you will get exactly that behaviour.
- Don't set the objective below current performance plus headroom. A 99th-percentile-latency objective the service has never met burns the budget permanently and trains everyone to ignore it.