1,480 Alerts, a 99.9% SLO, and Three-Second Checkout Delays
A payments service exposes a REST API and depends on a third-party fraud-check API. Over six weeks, the on-call team received 1,480 alerts; only 41 required action. The noisiest alerts are CPU above 60% for five minutes, a single 5xx response, and third-party latency above 800 ms. The service has a 99.9% monthly availability SLO but no error budget policy. Traces are sampled at 1%, and logs are retained for seven days. Customers report intermittent three-second checkout delays during peak hours, but dashboards show normal average latency. The team has 10 engineers, a weekly on-call rotation, and a change freeze during month-end. The third-party API has a published 99.5% availability SLA but no status webhook.
1. Redesign the alerting strategy. Explain which alerts should page, which should become tickets or dashboards, and how you would measure whether the change worked.
Read the worked answer
Classify alerts by action. Page only for user-visible symptoms or fast error-budget burn. A single 5xx is not actionable; it should be an event or a ticket if it repeats. CPU above 60% is a cause metric and should not page unless it correlates with saturation and SLO burn. Third-party latency above 800 ms should page only when checkout success or latency is affected; otherwise it belongs on a dependency dashboard.
Use multi-window burn-rate alerts. For a 99.9% SLO, page when the one-hour burn rate is high and the five-minute window confirms it, and send a ticket for slower burn. This reduces flapping and catches real outages.
Route by ownership. Alerts for the fraud provider go to the integration owner with a runbook that includes fallback behavior. Infrastructure alerts go to the platform queue, not the payments on-call.
Target. From 1,480 alerts to a small page set, perhaps under 20 pages per month, with every page linked to a runbook and an SLO. Measure precision: the fraction of pages that required action. If precision stays low, delete or downgrade the alert.
Risk. Removing cause-based alerts can hide capacity problems; keep them as dashboards and tickets, not pages.
2. Define the SLO and error budget policy. How would you treat the third-party fraud dependency, given its weaker SLA?
Read the worked answer
Define the SLI at the customer boundary: successful checkout requests within a latency threshold divided by valid checkout requests, measured over a rolling 30-day window. The current 99.9% availability SLO is a start, but availability without latency hides the three-second delays. Add a latency SLO, for example 99% of requests under one second, and track them separately.
Error budget. At 99.9%, the monthly budget is about 43 minutes. Write a policy: when the budget is exhausted, releases that affect checkout require approval and focus shifts to reliability; when burn is fast, page. The policy must name who can override and what evidence is required.
Dependency treatment. Measure the third-party call separately and end-to-end. Do not exclude fraud-check failures from the user SLO if checkout fails; instead use the internal attribution to decide whether to fail open, queue, or degrade. The third party's 99.5% SLA is weaker than your SLO, so a fallback decision is required.
Evidence. Publish SLO reports weekly with burn rate and incidents. If the team cannot measure a latency SLI today, add histogram metrics before changing the policy.
3. Diagnose the intermittent three-second delays. What telemetry would you add, what hypotheses would you test, and what would make you stop the investigation?
Read the worked answer
Why averages mislead. A few slow requests can produce three-second checkout delays while the mean stays normal. Start with histograms and tail percentiles by endpoint, region, customer, and dependency. Break down the request path: queue time, fraud-check time, database time, and retry count.
Increase observability temporarily. Raise trace sampling from 1% to 10-20% during peak for a bounded period, and extend log retention beyond seven days for checkout errors. Add structured fields for fraud-provider latency, timeout, retry, and circuit-breaker state. Ensure clocks are synchronized.
Hypotheses to test. Peak-hour delays may come from connection-pool waits, retries amplifying load, or the fraud provider's tail latency. Compare slow traces with pool wait time and retry counts. Run a controlled experiment: disable retries for a small canary or add a fallback response, and observe tail latency. Do not change production broadly without a rollback.
Decision. If slow traces cluster at the third-party call, add a timeout and fallback with a business rule. If they cluster at the database, fix pooling and query plans. Keep the increased sampling until the cause is confirmed, then set a sustainable level based on cost.