Alert Fatigue and a Missed Latency SLO
An e-commerce platform runs 60 services on IBM Cloud. IBM Cloud Monitoring receives about 1,200 alerts per week; 82% are closed by the on-call engineer without a change, and 31% are duplicates from the same root cause. Last Tuesday the product search p95 latency exceeded its 800 ms SLO for 40 minutes, but no one was paged. The only related alert was a CPU-high alert on one pod; it fired after 25 minutes and was silenced because it had already fired three times that day. Each service has six dashboards, and logs are shipped to IBM Log Analysis at 4 TB per day with no sampling. Traces are collected for only two services. The team wants to reduce pages and catch customer-facing failures faster without increasing spend significantly.
1. Diagnose why the existing alerting missed the search latency breach and propose a symptom-based alerting design for the SLO.
Read the worked answer
The CPU alert is a cause-based signal. One busy pod does not prove that customers see slow search, and silencing it removed the only related notification. The p95 SLO breach had no alert because alerting was organized around infrastructure metrics rather than the user-visible symptom. Start from the SLO: page when the search p95 latency burn rate threatens the budget, using request latency measured at the load balancer or ingress and confirmed by application metrics. Use multiwindow burn rates: a fast burn over 1 hour and 5 minutes, a slower burn over 6 hours and 30 minutes, and a ticket for a 3-day burn. Keep CPU and memory alerts as diagnostic context on the dashboard, not as pages unless saturation is itself customer-impacting. Give each service one primary dashboard that shows the SLO, burn rate, error rate, latency, and saturation, and attach a runbook that names the first three checks.
2. Recommend a prioritized observability data plan given the 4 TB/day log volume and limited tracing. What would you keep, sample, or drop?
Read the worked answer
Prioritize data by its ability to answer customer-impact questions. Keep all error logs, slow-request logs, security and audit events, and change or deployment events. These are low-volume relative to debug logs and are hard to reconstruct later. Sample high-volume debug and info logs aggressively, such as 1-5%, and use structured fields so sampling still supports aggregation. Use tail-based tracing for at least the customer-critical journeys, keeping all traces with errors or latency above the SLO threshold and sampling successful traces at 10-20%. Export long-term logs to IBM Cloud Object Storage instead of keeping everything in the hot log index. Control metric cardinality by dropping unbounded labels such as user ID or request ID from metrics; keep those in logs or traces. This plan reduces hot storage while improving the chance of catching the next latency incident.
3. How would you roll out and validate the new alerting without creating a coverage gap?
Read the worked answer
Run the new SLO alerts in shadow mode alongside the existing alerts first. Send them to a separate channel and compare them with real incidents, customer reports, and the old pages for at least two weeks. Backtest against the last 90 days of metrics to estimate how many pages the new rules would have produced and whether they would have caught the search latency breach. Define precision and recall targets, for example at least 70% of pages actioned and no missed SLO breach in the backtest. Write runbooks and have on-call engineers review the rules before cutover. Then switch page delivery service by service, keep the old rules as non-paging context for one more week, and review every new page in the weekly operations review. Do not remove the old alerts until the new ones have demonstrated coverage.