Brute Force or Clock Drift?
You are tuning a Splunk ES correlation search called 'Brute Force Followed by Successful Logon'. It has fired 380 times in the last 20 minutes across three member servers: APP-03, SQL-02 and file server FS-01. The failed-logon source is 10.20.4.0/24, which your threat-intel feed labels as an internal vulnerability scanner. You also notice indexer idx-02 reporting an NTP offset of +47 minutes since 03:00, while the search uses a 5-minute correlation window. One of the matched 'successful logon' events is EventCode 4624 Type 3 for account svc_backup on SQL-02 at 09:14; svc_backup normally authenticates only from 10.20.9.0/24, and a daily 09:14 backup job from 10.20.4.55 has run for two years. The night shift has two analysts and 40 other open alerts.
1. Is this a real brute-force compromise or a data-quality artefact? How do you establish the true timeline, and what would change your verdict?
Read the worked answer
Treat 380 alerts as one hypothesis to test, not 380 incidents.
Establish the true timeline before interpreting sequence. Pull both _time (parsed event time) and host-reported time for each sourcetype. idx-02's +47 minute offset means events from FS-01 may be indexed after events they actually preceded. A 47-minute skew is nine times the 5-minute correlation window, so any 'failure then success' pairing that crosses indexers is meaningless until corrected. Re-run the search with per-host clock correction, or first restricted to a single indexer, and see whether the pattern survives. Often it evaporates.
Then test the match itself. svc_backup at 09:14 from 10.20.4.55 matches a two-year daily baseline: that success is scheduled backup work, not a compromised password, and the scanner subnet label confirms it. Check logon type (3 network, 2 interactive, 10 RDP), authentication package, and whether the source port falls in a scanner's ephemeral range rather than a fixed service port.
What would flip me to a compromise: a 4624 for svc_backup from outside 10.20.4.0/24 and 10.20.9.0/24; a success on a host with no scheduled job, seconds after failures; a logon type the account never uses; or failures against accounts that do not exist in the domain (password spraying rather than a scanner). Absent those, this is scanner traffic plus clock skew - a pipeline defect, not an intrusion.
2. What do you fix in the rule and the pipeline, and what must you deliberately not exclude?
Read the worked answer
Fix the pipeline first, because a tuned rule on skewed data is still wrong. Remediate NTP on idx-02 via GPO or w32tm, add offset alerting so a drift over 60 seconds pages someone, and require hosts to report their own time so correlation never depends on indexer arrival order.
Fix the rule structurally. Rebuild the sequence over per-host corrected time with a window wide enough to be meaningful (15 minutes, not 5) and require the success to come from the same source address that produced the failures. That single condition would have killed all 380 of these matches.
Exclude narrowly. Exclude the specific scanner service accounts (svc_nessus, svc_qualys or equivalent) from this rule, and only from this rule. Do not exclude 10.20.4.0/24. An attacker who lands on a scanner host inherits that address, and a subnet-wide exclusion hands them a blind spot in exactly the place you were watching. Also do not exclude svc_backup globally - its 09:14 behaviour is legitimate, but first-seen source addresses for service accounts are a genuinely valuable detection.
Keep the anomaly you found. Replace the suppressed logic with a rule for 'service account authenticates from a source address not seen in the prior 30 days', which is a far better use of the same data and will catch the real version of what this alert pretended to be.
3. With two analysts and 40 other open alerts, how do you triage this and what do you hand over at shift change?
Read the worked answer
Do not work the 380 alerts. Collapse them into one ticket and spend the time on the residual signal.
Time-box: 20 minutes on the correlation validation (timeline correction, logon-type check, source-address check). If the matches collapse as expected, downgrade to a data-quality ticket, not a security incident. That is the correct outcome and should be recorded as such.
The one thing worth escalating is the svc_backup source deviation, and only if it survives the corrected timeline. If it is genuinely the 09:14 job, close it with the baseline evidence attached so the next analyst does not redo the work.
Handover note must contain: what fired, the NTP offset on idx-02 and when it started, that the rule's 5-minute window is invalid across indexers, the specific rule change requested, and the residual risk accepted (service-account logons from the scanner subnet are now unmonitored until the new rule ships). Named owner and a date. A handover that says 'investigated, likely false positive' with no reasoning is how the same 380 alerts return tomorrow night.
Do not spend the shift building a 380-row spreadsheet. On a two-analyst night, the highest-value output is a fixed pipeline and a better rule, not a documented false positive.