I85PREP
⌘ K
Splunk Core Certified User study materials
Splunk Core Certified User study resources

Four Minutes Versus Fifteen Seconds

Splunk Core Certified User · Put your knowledge to work in context.

All case studies

Four Minutes Versus Fifteen Seconds

Priya runs index=web sourcetype=access_combined status=500 | stats count by uri over the last 7 days and it takes four minutes. A teammate runs index=web sourcetype=access_combined | stats count by status over the same period in 15 seconds. Priya then tries index=web *timeout* over 30 days to catch any error wording she can think of, and that search never returns. The web index holds roughly 900 million events over 7 days, there are a dozen concurrent users on the search head, and no data models are accelerated.

1. Which of Priya's two searches is expensive for reasons unrelated to how many events match, and what is Splunk doing differently in each case?

Read the worked answer

Both are expensive for extraction, but the second is the pathological one.

The *timeout* search. A leading wildcard cannot be resolved from the index's term dictionary. Splunk cannot jump to matching terms; it must retrieve and evaluate the text of every event in the 30-day window. That is a full scan of roughly four times the 900M events, with a regex-like match on each. Over 30 days with no acceleration, that is not a slow search, it is an unbounded one.

The stats count by uri search. It scans the same events as the teammate's 15-second search, but status and uri are search-time extracted fields. The teammate groups on status, which has a handful of distinct values; Priya filters on status=500 (meaning status must be extracted for every event, because it is not an indexed field) and then groups on uri, which is high-cardinality - hundreds of thousands of distinct values to build and sort. Extraction cost plus cardinality is the difference, not the number of matching events.

The lesson the exam cares about: only index, host, source and sourcetype (and explicitly indexed fields) reduce the data Splunk reads. Everything else is processed after retrieval.

2. Name three changes to Priya's first search or its environment that would most reduce its runtime, and explain why each helps.

Read the worked answer
  • Accelerate and use tstats. Build a data model over the web data, accelerate it, and query the summaries: | tstats count from datamodel=Web.Web where Web.status=500 by Web.uri. This reads precomputed summary rows instead of raw events, removing both the scan and the extraction cost. This is the single biggest win and it is reusable by the whole team.
  • Set the search mode deliberately to Fast. In Fast mode Splunk does not perform field discovery and only extracts fields the search actually references, which cuts extraction work substantially on a wide event set. Smart mode (the default) adds discovery overhead Priya does not need once her search is written.
  • Narrow time and use a summary index for the daily view. If the business question is 'top slow URIs today', run the expensive aggregation once on a schedule into a summary index and let the report read the small summary. Failing that, reduce the range to 24 hours and add earliest/latest explicitly.

Also worth doing: alias or index status as an indexed field if this filter is used constantly, and stagger the other users' scheduled searches so they do not compete for the same search head at the same minute.

3. Priya rewrites the search and it is still slow. How does she find out which part is actually costing the time, and what should she look at?

Read the worked answer

Use the Job Inspector (the Job menu > Inspect Job, or the 'i' icon on the search job). It breaks down the run into phases and reports execution costs per command.

What to look for:

  • Execution costs per command - a single stats, sort, dedup or transaction dominating usually means cardinality or memory, not I/O.
  • Search job properties - scanCount versus resultCount. A huge scanCount with small resultCount means the base search is not filtering early enough; a huge resultCount means too much data is being passed down the pipeline.
  • Messages - 'Results may be incomplete', memory limits, or skipped search due to concurrency limits all show here rather than in the results.
  • Duration breakdown - time in queue versus time executing tells her whether the search head is simply saturated by the dozen concurrent users.

Her first move should be to confirm the bottleneck is in the base search rather than in the reporting command, because the fixes are completely different: filtering earlier and narrowing time for the base, versus replacing sort/dedup with stats for the reporting stage.