The Alert That Fired 4,000 Times
The SIEM rule "Multiple Failed Logons" generates roughly 4,000 alerts per day. Triage shows 92% come from the vulnerability-scanner service account, 6% from help-desk password resets, and 2% from a recurring set of 30 hosts spread across the week - one of which is a finance workstation. Tier-1 analysts handle about 40 alerts per shift, so in practice every alert is closed as benign without investigation. Full packet capture is retained 24 hours, NetFlow is retained 7 days, and Windows security event logs are retained 30 days. The SOC manager wants alert volume down before next quarter, and the IR lead wants assurance that a real password-spraying campaign would still be caught. Budget exists for either longer PCAP retention or one new log source, not both.
1. Reduce the noise from 4,000 alerts to a defensible volume. State exactly what you suppress or tune and what you keep.
Read the worked answer
The trap is "suppress the noisy account." If you exclude the scanner service account by name, you have just built a blind spot around an account that has broad read access and is frequently over-privileged.
Tune at the correlation layer, not by exclusion:
- Scanner: suppress failures only when source IP is in the scanner's known range and the account is the scanner account. Add a new rule: "scanner account failing or succeeding from any IP outside the scan range." That is a small number of events and is the one that means compromise.
- Help desk: suppress failures that match the expected pattern (from the reset jump hosts, targeting users with an open reset ticket). Keep failures from the help-desk VLAN against privileged accounts.
- Keep unconditionally:
- failures across many distinct accounts from one source (horizontal spray),
- failures against one account from many sources (distributed spray),
- any success immediately following a burst of failures on the same account,
- authentication from outside normal hours or geographies,
- failures against privileged accounts at any volume.
Add rate/entropy logic rather than a flat threshold: count distinct accounts per source IP and distinct source IPs per account over a window. Volume-based thresholds fail against low-and-slow; diversity-based detection is what catches it.
Target: a few hundred alerts a day that tier-1 can actually read. An alert nobody investigates is not detection - it is paperwork.
2. The 2% set of 30 recurring hosts - what pattern is that, and what do you do with it?
Read the worked answer
It is probably benign, and that is exactly why it is the most important thing on the board.
Two candidates:
- Configuration drift - a stale password on a scheduled task, a mapped drive, a service account on a legacy app, a mobile device retrying old credentials. Fails forever, never succeeds, nobody notices.
- Low-and-slow brute force or spray - deliberately kept under rate thresholds, which is precisely the adversary behaviour your proposed volume reduction would hide.
The discriminator is outcome: does a success ever follow the failures? A configuration problem produces failures forever and then a success on the day someone fixes the password - a burst of failures followed by success from the same source. Brute force looks the same on the wire and different only in whose account wins.
Actions:
- For each of the 30, pull the source IP, the process (via EDR/Sysmon), the target account, and whether it ever succeeded.
- Cluster them: if all 30 fail against a handful of service accounts from server subnets, it is drift. If they fail against many user accounts, it is spray.
- Close the drift properly - fix the stale credential, don't allowlist the host. Allowlisting a host that fails auth is how you hide the day it becomes a real attacker.
- Prioritise the finance workstation. Not because failure there is more likely malicious, but because the consequences of a missed compromise are worse. Same alert, different stakes - that is a triage decision, not a rule decision.
3. The IR lead needs proof that a real spraying campaign would still be caught after tuning. How do you prove it, and does the new log source or the longer PCAP retention matter more?
Read the worked answer
Prove it with controlled replays, not assurances. Before deploying, run approved adversary emulation against the tuned rules:
- spray from a single source against 50 accounts (slow, one attempt each),
- vertical brute force against one account,
- distributed spray from many sources against one account,
- a success after a failure burst.
Each must fire, and note the time-to-alert and the field that made it fire. Then report detection coverage as a metric - "known-bad patterns caught: 4/4, median 90 seconds" - instead of reporting only the alert count. The SOC manager's target and the IR lead's requirement are then the same number, viewed from two sides.
On the budget question: take the new log source.
Reasoning: 24 hours of PCAP is enough to triage an alert that fires today, which is the window tier-1 already works in. It is useless for the retrospective question - "was this host talking to that address three weeks ago?" - and that is where a spray that succeeded quietly actually gets found. But you already retain 7 days of NetFlow and 30 days of Windows security logs, so you have connection-level and authentication-level history. The marginal coverage from more PCAP is smaller than the gap from having no host-level telemetry.
The new log source should be chosen to close the biggest hole - typically EDR/Sysmon process-creation (turns "an account failed" into "this binary failed") or VPN authentication logs (spray against remote access, which the DC logs never see). Pick based on which one your detection gaps actually touch.