I85PREP
⌘ K
Splunk Core Certified User study materials
Splunk Core Certified User study resources

380 Emails and a 90-Second Dashboard

Splunk Core Certified User · Put your knowledge to work in context.

All case studies

380 Emails and a 90-Second Dashboard

Meridian's SOC has an alert called 'Failed Login Spike' scheduled every 5 minutes that emails 40 recipients. During a brute-force attempt it fired 380 times in two hours and the mail queue backed up. The SOC also has a 'Login Activity' dashboard with 12 panels, each running its own full search over the last 7 days; it takes 90 seconds to load and several panels time out. Finally, a scheduled report that feeds a compliance spreadsheet stopped running two weeks after its author moved to another team. There is one search head and no accelerated data models.

1. Describe the alert settings that would have prevented the 380 emails, and what you would change about the alert's structure.

Read the worked answer

Settings first:

  • Throttling. Enable throttling with both a time window ('suppress triggering for' e.g. 1 hour) and a suppress-results-by field (for example src), so the same offending source does not re-notify on every run. Without field suppression, a single ongoing attack regenerates an email every 5 minutes.
  • Trigger condition. Change from 'number of results > 0' to a threshold that means something, e.g. 'number of results > 20', evaluated on an aggregated result set rather than per raw event.

Structure change: the alert should fire once with a summary, not once per event. Make the search produce one row per source:

index=auth action=failure
| stats count by src, user
| where count > 20

That turns 380 near-identical triggers into one alert listing the top offenders. Also add a schedule window so the 5-minute search does not pile up on a busy search head, and confirm all 40 recipients are on a distribution list rather than individual addresses so the list can be managed in one place.

2. How would you restructure the dashboard to cut load time, and what would you check afterward to prove the improvement?

Read the worked answer

Collapse the 12 independent searches into one base search plus post-process searches.

  • The base search runs once, over the shortest useful range, returning only the aggregated fields the panels need - for example index=auth | stats count by user, src, action, _time-style summarization rather than raw events.
  • Each panel then runs a small post-process search (| stats sum(count) by user, | timechart ...) against that materialized result set in memory. Twelve panels then cost one scan, not twelve.
  • Reduce the time range where the panel does not need 7 days, and set panels that rarely change to only refresh on demand.
  • Longer term: accelerate a data model and serve the panels with tstats, which removes raw-event scanning entirely.

Proof: use the Job Inspector before and after to compare scanCount, per-command execution cost, and total duration; also check the number of search jobs dispatched per dashboard load in the search head's job listing. If the base search still dominates after restructuring, the problem is the time range and cardinality, not the panel count - the inspector will show that directly.

3. Why did the compliance report stop running when its author changed teams, and what are two durable fixes?

Read the worked answer

Scheduled reports run as their owner, with the owner's permissions. When the author moved teams, the role change very likely removed access to the app where the report lives, or to the index it reads - or the account was disabled. The scheduler then fails the job (permission denied, or the search is skipped/disabled), and because nothing alerts on scheduler failures, nobody notices for two weeks.

Durable fixes:

  1. Reassign ownership to a service account that is not tied to an individual, with a role that has exactly the read access the search needs, and share the report at app level so the team can maintain it.
  2. Make failure visible - enable a scheduled-search failure notification to the team, and review the Searches & Reports list periodically for jobs with a stale 'last run' or a future-looking 'next scheduled time'; check splunkd.log for scheduler errors.

Also verify the report's time range is relative (e.g. earliest=-1d@d) rather than absolute dates, so it keeps producing correct output once it is running again - an absolute range is the other classic reason a long-lived scheduled report silently returns the same stale window.