I85PREP
⌘ K
Splunk Power User study materials
Splunk Power User study resources

The Alert That Quietly Dropped Half Its List

Splunk Power User · Put your knowledge to work in context.

All case studies

The Alert That Quietly Dropped Half Its List

A SOC analyst owns a saved search that flags logons by terminated staff: index=wineventlog EventCode=4624 | search [search index=hr sourcetype=hr:terminations action=terminated earliest=-90d | fields user | format]. HR recorded 18,400 terminations inside the retention window. The analyst expects roughly 40,000 matching logons, but the alert returns 22,140. The job finishes in 8 seconds with no error banner, the HR subsearch returns all 18,400 rows when run on its own, and an identical pattern built on a 2,000-user department list produces totals that match expectations exactly. The alert is the only automated gate on the offboarding checklist and the next audit is in two weeks.

1. Explain why the 18,400-user subsearch loses data while the 2,000-user version does not, and name the limits responsible.

Read the worked answer

Two independent caps sit on every subsearch, and neither surfaces as an error:

  • Result cap (maxout) - default 10,000 rows. The 18,400-row HR list is cut at 10,000 before it is ever handed to the outer search; the 2,000-row department list fits comfortably.
  • Runtime cap - default 60 seconds. Not the trigger here (8s job), but it is the other silent truncator and should be ruled out.

After truncation, format collapses the surviving rows into one long (user=a OR user=b OR ...) string that is appended to the outer search. Nothing downstream knows terms are missing, so the search simply matches fewer events.

How to confirm: open the job inspector for the saved search and read the expanded search string stored on the job - you will see the OR-list stop short of the expected membership, or a subsearch result count near 10,000. Running | format maxout=25000 on the same subsearch and watching the count jump is the definitive test.

A 10,000-term OR list also means a huge search string and a parse-heavy search, so this design was fragile even below the cap.

2. Rewrite the detection so the membership test cannot be truncated, and state what you would verify after deploying it.

Read the worked answer

Move the roster out of the subsearch and into a lookup:

  1. Scheduled search (hourly) refreshes the roster and normalizes it:
index=hr sourcetype=hr:terminations action=terminated earliest=-90d
| stats latest(_time) AS term_time BY user
| eval user=lower(trim(user))
| outputlookup terminated_users.csv
  1. Detection:
index=wineventlog EventCode=4624
| eval user=lower(trim(user))
| lookup terminated_users.csv user OUTPUT term_time
| where isnotnull(term_time)

Why this is better:

  • No maxout ceiling: lookups are not subsearches, so 18,400 (or 180,000) rows behave the same.
  • The search string stays short and the lookup table is reused by every detection that needs it.
  • lower(trim(...)) on both sides is mandatory - lookup matching is exact and case-sensitive, and roster exports routinely carry mixed case and trailing whitespace.

Post-deploy verification: compare | inputlookup terminated_users.csv | stats count against the HR source count; run the new detection and the old one with maxout=25000 side by side and reconcile the difference, so you can explain the historical undercount to the auditor rather than just closing the ticket.

3. A colleague proposes keeping the subsearch but bumping it to `| format maxout=25000`. What problems remain?

Read the worked answer

Raising maxout fixes the row count, not the design:

  • Search string size. 18,400 OR terms produce a search string on the order of hundreds of kilobytes. Parse time balloons, the search string may hit length limits, and the saved-search definition becomes unreadable in the UI.
  • Cost per run. The OR-list is re-evaluated on every execution; a lookup is a single hash-style join against a reusable file.
  • Reuse. Every other detection needing the same roster copies the pattern and its limit.
  • Silent regression. Nothing alerts you when the roster grows past 25,000 next quarter - you are back to today's ticket.

Also note maxout is only one of the two caps; if the subsearch ever slows down, the 60-second runtime limit truncates it regardless of maxout. Use the subsearch limit bump only as a temporary diagnostic to prove the size of the historical gap, then replace it with the lookup.