The Alert Flood
A SOC receives 3,200 alerts per day from EDR, IDS, and cloud logs. Two analysts triage them. The team's mean time to acknowledge is 9 hours, and 40% of alerts are known false positives from a vulnerability scanner and a backup job. Last month a real intrusion - lateral movement via stolen credentials - sat in the queue for 26 hours before being noticed. Management wants 'more detection coverage' and is considering buying another tool. The SOC has no documented triage playbook and no alert-tuning process. The intrusion's initial alert was a medium-severity 'unusual logon' event.
1. Why is buying another detection tool the wrong first move, and what should be done instead?
Read the worked answer
The SOC's problem is not detection coverage; it is signal-to-noise and triage capacity. With 3,200 alerts/day, two analysts, and 9-hour acknowledgement, adding a tool increases volume and worsens the bottleneck. The evidence: 40% known false positives are still firing, and a real intrusion waited 26 hours. The first moves are:
- Suppress/tune the known noise (scanner and backup) - this is free capacity.
- Document a triage playbook with severity definitions and escalation paths.
- Prioritize by risk, not by raw severity label - the medium 'unusual logon' was the real signal.
- Measure alert volume, false-positive rate, MTTD/MTTA, and time-to-escalate. Only after the pipeline is under control does additional coverage make sense - and then it should be targeted at gaps the playbook reveals.
2. Design a triage approach that would have caught the lateral movement sooner.
Read the worked answer
Use correlation and risk-based prioritization:
- Correlate identity + endpoint + network: an 'unusual logon' becomes high priority when it is followed by new host-to-host connections or credential use from an unusual source. A SIEM correlation rule on that sequence would elevate it.
- Baseline and flag deviation: stolen-credential lateral movement often looks like valid credentials used from a new source or at an odd time; alert on that pattern.
- Tiered queue: separate 'needs human judgement' from 'auto-closeable noise'; route high-risk correlations to immediate review.
- Playbook: for identity alerts, steps are: verify source, check for follow-on connections, check for new service creation, escalate if any hit.
- Feedback loop: every closed alert updates tuning; every miss becomes a detection-engineering ticket. This is process and correlation, not a new sensor.
3. What metrics would you report to management to justify the investment in process over tools?
Read the worked answer
Report the metrics that show the bottleneck moving:
- Alert volume and % auto-suppressed/tuned (should rise then stabilize).
- False-positive rate by source.
- MTTA and MTTD, and time-to-escalate for confirmed incidents.
- Escalation accuracy: % of escalations that are true positives.
- Coverage of playbooks: % of alert types with a documented playbook.
- Analyst load: alerts per analyst per shift. Tie these to outcomes: 'time-to-escalate for identity alerts dropped from 26h to X.' That is the business case for tuning and playbooks, and it also tells you when adding a tool would actually help.