Alert volume hides the missing 'why' in checkout latency
A payments platform runs 220 services on Kubernetes and has 38 Grafana dashboards, 410 alert rules, and a 99.95% availability SLO. Over the last quarter the on-call team received 46 pages per week; post-incident review classified 31 as actionable and 15 as duplicate or self-resolving. Mean time to repair is 94 minutes, and the top recurring incident is "checkout latency > 2s" affecting 0.8% of sessions. Dashboards show CPU, memory, request rate, error rate, and p99 latency per service, but no trace or span data. When checkout slows, engineers open five dashboards, compare timestamps by eye, and often escalate to the database team even when the database is healthy. Product owners want fewer pages and faster diagnosis before the holiday peak, when traffic will double. The platform team has budget for one quarter of engineering work and must justify whether to add tracing, improve alert quality, or both.
1. Using the three pillars, explain why the current monitoring stack cannot answer why checkout latency rises. Recommend a first-quarter investment and state the key trade-off.
Read the worked answer
The stack is metric-centric: it answers what changed (p99, CPU, error rate) but not where inside the request or why. Metrics are pre-aggregated, so by the time p99 crosses 2 seconds the specific slow dependency, retry loop, lock, or garbage-collection pause has been flattened away. Logs exist per service but are not correlated to a request, and 38 dashboards force engineers to align timestamps by eye. Traces supply causality: one trace shows parent-child span timing and attributes such as db.statement, http.status_code, and peer.service, so the slow hop is visible rather than inferred.
A defensible first-quarter investment is a narrow tracing rollout plus alert reform, not a broad dashboard project. Instrument the checkout path with OpenTelemetry, propagate context across HTTP and gRPC, and attach exemplars from latency histograms to traces so a p99 spike can be opened directly. At the same time, replace the top five symptom alerts with SLO burn-rate pages and demote low-severity alerts to tickets.
The trade-off is cost and coverage: tracing 100% of traffic adds SDK overhead and backend cardinality, while low head sampling will miss the 0.8% of affected sessions. Use tail sampling at a gateway to retain all error and slow traces plus a 10% baseline. Tracing alone would shorten diagnosis but leave the 46-page noise; alert reform alone would reduce pages but not explain the remaining incidents.
2. The team proposes deleting all 15 self-resolving alerts. Evaluate this proposal against observability principles and SLO burn-rate alerting.
Read the worked answer
Deleting all 15 is a risky overcorrection. The fact that an alert self-resolves does not prove it is useless; it may be a leading indicator of saturation, certificate expiry, or a dependency that recovers only because retries absorb the failure. The review classified the alerts as duplicate or self-resolving, not as non-actionable. The right test is customer impact and decision value: would a responder do something differently within the next 30 minutes?
A better approach is to separate paging from ticketing. Page only on multi-window SLO burn that threatens the 99.95% target quickly, for example 2% of the monthly budget consumed in one hour, and send slow-burn conditions to a ticket queue. Keep capacity and saturation alerts as tickets with a delay, and keep security or compliance alerts out of this cleanup entirely. For each of the 15, record the last time it preceded a real incident; retain those that are leading indicators and delete only those that duplicate another signal or have no owner.
Measure page precision after the change: pages per week, percentage actioned, and MTTR. Deleting all 15 might reduce noise but could also remove early warning and push the organization back toward reactive monitoring.
3. Define the trace data and success measures the platform team should use to prove the quarter worked.
Read the worked answer
The trace data should be deliberately scoped, not exhaustive. Capture service.name, deployment.environment, http.route, rpc.method, db.system, peer.service, http.status_code, and error.type; record retries as span events or child spans with attempt numbers; and include deployment version so a trace can be tied to a release. Propagate W3C Trace Context through all synchronous and asynchronous boundaries on the checkout path. Use tail sampling to keep 100% of error and slow traces plus a 10% baseline, and enforce attribute limits to control cardinality.
Success measures should connect telemetry to operational outcomes: MTTR reduced from 94 to under 60 minutes; pages reduced from 46 to under 15 per week with a higher actioned percentage; at least 70% of checkout incidents diagnosed from a trace rather than by dashboard comparison; trace-to-log correlation available for the top five services; and no more than a 15% increase in telemetry cost. Add a guardrail: the percentage of checkout requests with complete trace context must exceed 95%. These measures test whether the investment improved diagnosis, not merely whether traces exist.