Reducing alert noise without hiding a file-descriptor failure
An operations team for a ticketing platform has 14 services, 210 alert conditions, and 120 pages per week. Seventy percent of pages come from six conditions: CPU high on autoscaling batch nodes, memory high on a search cluster, and database latency high repeated for each of five services. Incident Intelligence is enabled, but responders say incidents contain 8-20 alerts and they cannot tell which service is causal. In a recent outage, a search cluster node ran out of file descriptors, causing search latency, then checkout errors, then 11 pages. The team wants to use New Relic AI and anomaly detection to reduce noise by 80% before next quarter. The constraint is no reduction in detection of customer-facing failures; they have 30 days to show improvement. The SRE lead must sequence the changes.
1. Would you start by turning on anomaly detection for all 210 conditions? Justify the sequence you would use to reach the noise goal without missing the file-descriptor scenario.
Read the worked answer
No. Turning on anomaly detection everywhere would add change without fixing duplicate or low-quality conditions. First classify pages by customer impact and causal role. Collapse the five database-latency conditions into one condition scoped by service or workload, and group them through Incident Intelligence so responders see one incident with related entities. For autoscaling batch CPU, tag those nodes and suppress expected scale events only in that workload, not globally. For search memory, keep a static saturation threshold because memory pressure is a capacity limit, and add missing file-descriptor monitoring. Anomaly detection is appropriate for stable seasonal signals where a baseline is meaningful; it is not a substitute for a hard limit alert on file descriptors. Then measure pages per week, alerts per incident, and missed customer-facing incidents against a four-week baseline. The guardrail is a synthetic checkout and search canary plus a review of every Sev1/Sev2 that did not page. If noise falls but a customer-facing failure is missed, roll back the specific suppression, not the whole program.
2. How should the team model the search cluster file-descriptor failure so New Relic AI can correlate it with downstream checkout errors?
Read the worked answer
AI can only correlate signals that exist and are connected. Add a host or process metric for open file descriptors and the process limit, then alert at a percentage of the limit with a static threshold. Tag the metric with cluster, node, and service so it joins the search cluster entity. Confirm entity relationships from search to checkout through the service map and workload definitions. Add a log pattern or log-to-metrics signal for file-descriptor errors, and keep search latency and checkout error-rate alerts as separate symptoms. Incident Intelligence can then group the file-descriptor alert with search latency and checkout errors by timing and entity relationships. Do not expect it to infer the missing metric from CPU or memory. Test the model with a controlled fault: lower a canary process limit, confirm the alert fires, confirm it groups with search latency, and confirm the runbook points to the search cluster. If grouping does not occur, fix tags and entity relationships rather than adding more alert conditions.
3. What metrics would prove the change worked, and what guardrail would you set?
Read the worked answer
Measure both noise and detection. Track pages per week, alerts per incident, percentage of incidents with a single causal service, MTTD, MTTA, MTTR, and the number of customer-facing Sev1/Sev2 incidents detected by monitoring rather than by customers. Compare against the four-week baseline. Set an explicit guardrail: no increase in missed customer-facing failures, verified by synthetic checkout and search checks plus a weekly review of incidents that did not page. A useful target is fewer pages per incident and shorter time to identify the causal entity, not just an 80% page reduction. If pages fall but alerts per incident stays high, the grouping work is incomplete. If MTTD rises, the suppression or anomaly threshold is too aggressive. Publish the results with the guardrail so the team can defend the change to product and finance.