Splunk Certified Cybersecurity Defense Analyst study materials
Splunk Certified Cybersecurity Defense Analyst study resources

Zero Results on the Dashboard, 4,200 Events in the Raw Index

Splunk Certified Cybersecurity Defense Analyst · Put your knowledge to work in context.

All case studies

Zero Results on the Dashboard, 4,200 Events in the Raw Index

Your Splunk ES deployment ingests 240 GB/day against a 300 GB/day license. Windows Security events from 1,800 servers arrive through 40 universal forwarders, and the Endpoint and Authentication data models are accelerated with a 7-day summary range. On Monday the SOC's 'Failed Logon Spike' panel returns zero results for the last 4 hours, yet a raw search of index=wineventlog EventCode=4625 returns 4,200 events in the same window. The panel uses | tstats summariesonly=t count from datamodel=Authentication.Failed_Authentication by Authentication.user. Meanwhile the license dashboard shows a 92% rolling 30-day average and the wineventlog indexing queue is spiking. The shift lead wants to disable and rebuild acceleration for all data models immediately.

1. Lay out the diagnostic sequence that separates ingestion lag, CIM normalization failure, and acceleration failure. Name the specific searches or UI checks at each step and justify the order.

Read the worked answer

Work from the cheapest, least disruptive check outward, and always prove which layer is broken before touching acceleration.

  1. Is the data arriving, and arriving late? Compare _time with _indextime on the 4625 events. If events carry timestamps 30-60 minutes older than _indextime, the queue spike means the problem is delivery lag, not detection. This also explains the empty panel: summariesonly=t summarises what has been indexed, and a summary range that lags real time yields zero for the newest bucket.
  2. Is the data normalized? Run | datamodel Authentication Failed_Authentication flat (no tstats) over the same window. If it returns events, CIM tags and field aliases are fine and the problem is downstream. If it returns nothing, the sourcetype has lost its tag or the app/props config changed.
  3. Is acceleration healthy? Settings > Data Models > Authentication: check status (enabled vs error), percent complete, last build time, and whether the search is being skipped. A model that is 62% built or whose build was skipped produces partial, stale summaries.
  4. Cross-check with summariesonly=f over the same window. If that returns data but summariesonly=t does not, acceleration is the fault and the earlier steps were clean.

Only step 3 or 4 justifies a rebuild, and a rebuild of all models would consume search head capacity that the queue spike says you cannot spare.

2. The license is at 92% of a 30-day average and the wineventlog indexing queue is spiking. Name two actions you would take and two you would avoid, with reasoning for each.

Read the worked answer

Take:

  • Fix the ingestion path first. A spiking queue means indexers are the bottleneck; investigate per-index queue fill, parsing queue vs indexing queue, and whether a single forwarder or sourcetype is flooding. Detection accuracy is downstream of this.
  • Validate acceleration selectively. Reconcile one model against raw data over a short window (last 6 hours) rather than rebuilding 7 days. If the summary lags, a short-window backfill of the affected time buckets restores the panel without touching 240 GB/day of raw data.

Avoid:

  • A full 7-day rebuild of multiple data models during business hours. Acceleration rebuilds are themselves scheduled searches on the search head; they compete for the same concurrency slots as correlation searches and can silently cause detection gaps (a stale summary returns zero, and zero looks like safety).
  • Widening the tstats time range or running summariesonly=f across 7 days as a 'quick fix'. With the license at 92% and a live queue spike, that converts a display problem into an ingestion and capacity incident, and it does not fix the panel for future data.

A useful rule for this exam scenario: never repair a visibility problem by consuming the capacity that caused it.

3. Draft the message to the shift lead (under 100 words) stating what is and is not known, and what they should do in the next hour.

Read the worked answer

Something like:

The 4,200 failed logons are real and in the index; only our summary-based panel is blank. We have not yet proven whether this is index-time lag or a broken acceleration build, so treat the panel as unreliable for now. Do not open notables from that panel as 'no activity'. For the next hour: (1) triage 4625s directly with a raw search against the 4,200 events, (2) send me the top 5 sources by count, (3) hold any acceleration rebuild until I confirm. I will report back within 60 minutes.

The important judgement is separating absence of evidence from evidence of absence. The shift lead should keep working the raw data, and the panel should be labelled as degraded rather than interpreted as an all-clear. Note explicitly what is not known (root cause, how long the summary has been stale) so nobody over-reads the update.