One Rule Owns 78% of the Notable Queue
You run Splunk ES with risk-based alerting. The risk window is 24 hours and the aggregation threshold that creates a risk notable is 100. Over the past week the SOC produced 215 notables per day, of which 168 came from a single rule, 'Excessive Failed Logons'. That rule fires roughly every 10 minutes against about 300 backup and integration service accounts and writes risk_score=20 per execution to risk_object_type=user. A second rule, 'Suspicious PowerShell Encoded Command', writes risk_score=60 but fires only 2-3 times a day, and its notables sit unworked at the bottom of the queue. Asset and identity lookups are populated, but the service accounts in identities.csv have no category and no owner. Three analysts work an 8-hour shift and can realistically close about 40 notables per day.
1. Explain why this rule overwhelms the queue, then rank three remediation options (throttle the rule, lower its risk_score, restructure what it scores) and recommend one.
Read the worked answer
The arithmetic explains the queue: risk is summed per risk object over the 24-hour window, not across all objects. A single backup account hit every 10 minutes accumulates 144 executions × risk_score 20 = 2,880 risk, roughly 29× the threshold of 100. So nearly every one of those 300 accounts crosses the threshold independently, every day, from behaviour that is expected.
Ranked options:
- Restructure what the rule scores (best). Score only the behaviour that is actually anomalous: failed logons exceeding a per-account baseline, or a burst followed by a successful logon (4625 → 4624), or failures against privileged accounts. This preserves the rule's ability to catch misuse of a service account, which is a real attack pattern (credential stuffing against a service principal).
- Lower risk_score for this population via a risk factor multiplier (acceptable, second). A lookup-driven risk factor of ~0.05 for the backup category keeps the events in the risk index and keeps them searchable during hunts, while preventing constant threshold crossings.
- Throttle the rule (worst). Throttling suppresses the alert action for a key, and for a risk rule that includes suppressing the write to the risk index. You lose the accumulated context that makes RBA work, and the entity goes quiet even when its behaviour changes.
A blanket exclusion of the 300 accounts is tempting but is the same failure as throttling with extra steps: the account that suddenly starts authenticating from a new subnet becomes invisible.
2. The encoded-PowerShell rule is being starved of attention. Using RBA mechanics, how would you make that signal surface without simply raising its score to 500?
Read the worked answer
The point of RBA is that weak signals combine. Raising one rule to 500 just moves the noise problem to a different rule.
Better approach:
- Make the score proportional, not flat. Score 20 for an encoded command on a workstation and 60-80 when the same host has an open risk contribution from another rule, or when asset priority in the lookup is critical (multiply risk_score by priority before collection).
- Use threat objects. If the threat intelligence framework has the host's destination on a threat list, the threat object's risk lands on the same entity; a 60-point PowerShell hit that also touched a known-bad IP becomes a 100+ aggregate and crosses the threshold on corroboration rather than on a hard-coded number.
- Attach the right entity types. Ensure the rule writes both
risk_object_type=system(the host) andrisk_object_type=user(the executing account) so PowerShell risk on a host adds to authentication risk on that host. Many starved rules are starved because they only score one entity type and never accumulate. - Give the aggregation search a chance to see it. Check that the rule's schedule and the risk window overlap sensibly: a 2-3 per day rule with a 24-hour window is fine, but if the aggregation runs against a 1-hour window in practice, low-frequency rules can never reach the threshold.
Then measure: after the change, PowerShell-related notables should rise, and the analyst notes should show the risk came from more than one contributing rule.
3. The SOC manager proposes lowering the aggregation threshold from 100 to 60 to 'catch more'. What evidence would you require before agreeing, and what would you predict happens?
Read the worked answer
Require two pieces of evidence.
- A score distribution of known outcomes. Take 30 days of notables, label them true positive / benign by analyst disposition, and plot the aggregate risk score at closure. If true positives cluster at 140-200 and benign ones cluster at 100-115, the threshold is already too low and the answer is to raise it, not lower it. If genuine incidents are being found at 65-95 and closed as benign at 100+, the threshold is genuinely mis-set.
- Capacity math against the new volume. With 3 analysts × 8 hours and roughly 40 closable notables/day, a threshold change that pushes volume to 90/day does not 'catch more'; it converts two-thirds of the queue into unworked backlog, and the real incident ages out inside it.
The prediction: at threshold 60, the still-unfixed 'Excessive Failed Logons' rule floods further (every service account already clears 100, so 60 changes little there, but the long tail of two- and three-rule combinations now surfaces), volume roughly doubles, mean time to triage climbs, and analysts start bulk-closing notables. Lower the threshold only after the dominant rule is restructured and the score distribution shows a real gap between benign and malicious populations.