Alert Fatigue in a Three-Analyst SOC
A SOC has three analysts covering 07:00 to 19:00 with no overnight staff. The SIEM ingests 1,200 endpoints, 40 servers and 60 network devices and produces roughly 4,800 alerts per day. A ruleset inherited from a former managed-security provider generates 61 percent of them: multiple failed logons at a threshold of 5 in 10 minutes, new service installed, and outbound connection to a low-reputation IP. Last quarter the team closed 71 percent of tickets as false positive in under two minutes and missed a real ransomware precursor, a service installed at 02:14 by a domain administrator account, which sat in the queue nine hours. Hot storage keeps 14 days; cold storage keeps 90 days but is queried only during investigations. EDR is deployed on every endpoint, but its alerts are not forwarded to the SIEM. The CISO wants alert volume halved in 30 days without increasing mean time to detect.
1. Explain what the two-minute closures and the missed 02:14 service installation reveal about the program, and why halving volume alone will not fix it.
Read the worked answer
The two-minute closures are a behavior symptom, not a metric to be proud of. At 4,800 alerts across two shifts, roughly 300 alerts per analyst per day is more than can be investigated; the queue discipline that emerges is close-on-sight for anything matching a familiar pattern. That is rational under the workload and is exactly how a genuine signal gets closed.
The missed service install shows two distinct failures. First, there is no asset or identity context on the alert: nothing told the analyst that a domain administrator installing a service at 02:14 is unusual, so it looked like the other failed-logon noise. Second, there is no overnight coverage, so an alert created at 02:14 has no chance of being actioned before morning regardless of its quality. Reducing volume by 50 percent does not address either one; it just makes the queue shorter while the same detection gap remains.
Also note the measurement trap: the team is implicitly rewarded for fast closure and low open-ticket counts. Any tuning program has to change the success metric at the same time as the rules, or analysts will keep optimizing for closure speed.
2. Draft a prioritized 30-day plan that reduces noise without dropping detection coverage.
Read the worked answer
Week 1, measure and add context before deleting anything. Enable alert-to-asset and identity enrichment in the SIEM, forward EDR alerts in, and count what actually fires per rule per day. You cannot suppress safely until you know which rules fire on which asset classes.
Week 2, tune the top three noisy rules rather than disabling them. Failed logons: exclude service accounts and known vulnerability-scanner sources, raise the threshold on server subnets, and keep the rule at its original sensitivity for interactive logons on user segments. New service installed: keep the rule but split it into high-fidelity variants (service on a domain controller or server, install by a privileged account, install outside business hours) and low-fidelity variants that go to a weekly report. Low-reputation outbound: replace reputation-only matching with first-seen-destination plus EDR correlation, since reputation absence is not evidence of malice.
Week 3, implement risk-based routing: anything touching privileged accounts, domain controllers, or backup infrastructure pages immediately; the rest goes to a reviewed queue. Add an overnight triage path, even if it is an on-call rotation with a documented escalation threshold.
Week 4, validate. Re-run the historical ransomware precursor and confirm it now generates a routed alert. Publish volume, true-positive rate, and detection coverage together.
3. Which measures would you present to prove the change did not increase detection risk?
Read the worked answer
Volume alone is meaningless; pair it with coverage and latency evidence.
Detection coverage: replay a fixed set of representative attack behaviors against the tuned ruleset, ideally including the actual 02:14 service-install precursor and a handful of MITRE-mapped technique simulations. Report pass/fail per technique before and after.
Latency: track time from event to triage, not time from ticket open to close. The 02:14 case would have shown a nine-hour event-to-triage gap; that is the number the CISO should see trending down.
Fidelity: true-positive rate and false-positive rate per rule, plus the count of rules suppressed entirely. A rule suppressed without an owner and a review date is a liability; require both.
Queue health: age of oldest unreviewed alert and percentage of alerts untouched at 12 hours. That surfaces the overnight gap directly.
Retention: confirm that tuned-out events still land in cold storage for the 90 days, so suppressed alerts remain searchable during an investigation. Suppression at the alert layer is reversible; suppression at the collection layer destroys evidence, and that distinction should be stated explicitly when presenting results.