Zero Results on the Dashboard, 4,200 Events in the Raw Index
Your Splunk ES deployment ingests 240 GB/day against a 300 GB/day license. Windows Security events from 1,800 servers arrive through 40 universal forwarders, and the Endpoint and Authentication data models are accelerated with a 7-day summary range. On Monday the SOC's 'Failed Logon Spike' panel returns zero results for the last 4 hours, yet a raw search of index=wineventlog EventCode=4625 returns 4,200 events in the same window. The panel uses | tstats summariesonly=t count from datamodel=Authentication.Failed_Authentication by Authentication.user. Meanwhile the license dashboard shows a 92% rolling 30-day average and the wineventlog indexing queue is spiking. The shift lead wants to disable and rebuild acceleration for all data models immediately.
1. Lay out the diagnostic sequence that separates ingestion lag, CIM normalization failure, and acceleration failure. Name the specific searches or UI checks at each step and justify the order.
Read the worked answer
Work from the cheapest, least disruptive check outward, and always prove which layer is broken before touching acceleration.
- Is the data arriving, and arriving late? Compare
_timewith_indextimeon the 4625 events. If events carry timestamps 30-60 minutes older than_indextime, the queue spike means the problem is delivery lag, not detection. This also explains the empty panel:summariesonly=tsummarises what has been indexed, and a summary range that lags real time yields zero for the newest bucket. - Is the data normalized? Run
| datamodel Authentication Failed_Authentication flat(no tstats) over the same window. If it returns events, CIM tags and field aliases are fine and the problem is downstream. If it returns nothing, the sourcetype has lost its tag or the app/props config changed. - Is acceleration healthy? Settings > Data Models > Authentication: check status (enabled vs error), percent complete, last build time, and whether the search is being skipped. A model that is 62% built or whose build was skipped produces partial, stale summaries.
- Cross-check with
summariesonly=fover the same window. If that returns data butsummariesonly=tdoes not, acceleration is the fault and the earlier steps were clean.
Only step 3 or 4 justifies a rebuild, and a rebuild of all models would consume search head capacity that the queue spike says you cannot spare.
2. The license is at 92% of a 30-day average and the wineventlog indexing queue is spiking. Name two actions you would take and two you would avoid, with reasoning for each.
Read the worked answer
Take:
- Fix the ingestion path first. A spiking queue means indexers are the bottleneck; investigate per-index queue fill, parsing queue vs indexing queue, and whether a single forwarder or sourcetype is flooding. Detection accuracy is downstream of this.
- Validate acceleration selectively. Reconcile one model against raw data over a short window (last 6 hours) rather than rebuilding 7 days. If the summary lags, a short-window backfill of the affected time buckets restores the panel without touching 240 GB/day of raw data.
Avoid:
- A full 7-day rebuild of multiple data models during business hours. Acceleration rebuilds are themselves scheduled searches on the search head; they compete for the same concurrency slots as correlation searches and can silently cause detection gaps (a stale summary returns zero, and zero looks like safety).
- Widening the tstats time range or running
summariesonly=facross 7 days as a 'quick fix'. With the license at 92% and a live queue spike, that converts a display problem into an ingestion and capacity incident, and it does not fix the panel for future data.
A useful rule for this exam scenario: never repair a visibility problem by consuming the capacity that caused it.
3. Draft the message to the shift lead (under 100 words) stating what is and is not known, and what they should do in the next hour.
Read the worked answer
Something like:
The 4,200 failed logons are real and in the index; only our summary-based panel is blank. We have not yet proven whether this is index-time lag or a broken acceleration build, so treat the panel as unreliable for now. Do not open notables from that panel as 'no activity'. For the next hour: (1) triage 4625s directly with a raw search against the 4,200 events, (2) send me the top 5 sources by count, (3) hold any acceleration rebuild until I confirm. I will report back within 60 minutes.
The important judgement is separating absence of evidence from evidence of absence. The shift lead should keep working the raw data, and the panel should be labelled as degraded rather than interpreted as an all-clear. Note explicitly what is not known (root cause, how long the summary has been stale) so nobody over-reads the update.