I85PREP
⌘ K
Splunk Power User study materials
Splunk Power User study resources

The Pivot That Disagrees With the Search

Splunk Power User · Put your knowledge to work in context.

All case studies

The Pivot That Disagrees With the Search

A team built a data model over index=proxy sourcetype=proxy:access with a root event and a child constrained on status>=400, then accelerated it for a 12-month range. A Pivot panel counts errors by url_category and dest. The raw SPL equivalent, index=proxy sourcetype=proxy:access status>=400 | stats count BY url_category, returns 8–14% more events than the Pivot, and the most recent 15 minutes of the Pivot is always empty. Separately, url_category was added as a props.conf extraction last sprint, and the Pivot's url_category column is entirely null for events older than that change.

1. Explain the two discrepancies (the volume gap and the empty recent tail). How do you determine which side is wrong?

Read the worked answer

Two different mechanisms are at work, and neither means the raw numbers are wrong:

The empty tail = acceleration lag. Data model acceleration is built by a scheduled summarization job. Between cycles, the newest summaries do not exist yet, so a search that reads only summaries sees nothing for the last few minutes. This is expected behavior, not a bug.

The volume gap = model constraints differ from the raw search. The accelerated summaries only contain events that satisfy the model's own constraints. Likely culprits here:

  • status typed as a string in the model while the raw search compares numerically (or vice versa), so boundary/parse cases drop out.
  • Events where status is null or malformed - status>=400 in raw SPL treats a missing field as no-match, but the model's constraint may be applied differently at summary time.
  • The root event's search string is narrower than the raw search (extra sourcetype or index filters, or a lookahead that excludes some lines).

Arbitration procedure:

| tstats summariesonly=f count FROM datamodel=proxy.error_events BY _time span=1d

vs the same with summariesonly=t. Compare them day by day against the raw stats output. If summariesonly=f matches raw, the model's constraints are fine and you are seeing lag or summary gaps. If summariesonly=f still lags raw, the constraint itself is the problem and belongs in the model.

Also inspect the model's objects in the Data Model editor - constraint text, field types, and whether status is even a model field or only an inherited raw field.

2. Why is `url_category` null for older events in the Pivot even though the props.conf extraction works in ad hoc search?

Read the worked answer

A data model stores an enumerated field list, and summaries are precomputed at acceleration time. A props.conf extraction added later does not retroactively become a model field, and it certainly does not retroactively populate summaries that were written before it existed.

Three compounding facts:

  1. url_category was not in the model's field list when the older summaries were built, so those summary rows have no value for it.
  2. tstats can only report on fields that exist in the data model. Even summariesonly=f will not invent a field the model does not declare.
  3. Re-accelerating is not instant - history must be re-summarized, and until it completes, a rebuild looks identical to "still broken".

Fix sequence:

  1. Add url_category to the model (as a field on the relevant object, with a correct type - string, not number).
  2. Save the model, then rebuild the acceleration for the required range rather than waiting for incremental summarization.
  3. During the rebuild, verify with summariesonly=f so you can tell progress from failure.
  4. Only then re-point the Pivot and confirm old and new events both populate.

Going forward: any new field that a Pivot must group by has to be added to the model and followed by a rebuild. Treat props.conf changes and data model changes as two separate release steps with two separate verifications.

3. Leadership now wants a real-time alert on 5xx bursts. Should it use the accelerated data model? Justify with the tradeoffs.

Read the worked answer

No - not for the real-time alert.

  • Acceleration is batch. Summaries always trail the newest data by at least one job cycle, so an accelerated alert would fire late or not at all on the freshest burst - exactly the window that matters for incident response.
  • Real-time searches are expensive and cap-bounded. They run continuously and are limited by the number of concurrent real-time searches the platform supports; they are best reserved for a small number of genuinely time-critical detections.
  • Model granularity bounds accuracy. The summaries are computed at a grain (and with constraints) chosen for reporting. A burst detector needs raw event timing, not 5-minute buckets.

Recommended split:

  • Real-time / near-real-time alert: a narrow raw SPL search on index=proxy sourcetype=proxy:access status>=500, scheduled every minute is usually sufficient and far cheaper than true real-time.
  • Dashboards and trend reporting: keep using the accelerated model with tstats - that is where it earns its keep.
  • Reconciliation: keep the two in a documented relationship. Periodically compare the alert's raw count against tstats summariesonly=f for the same window so that the inevitable "dashboard says 400, alert fired 430" question has a prepared answer.

The judgment is not "acceleration good or bad" - it is that acceleration optimizes for fast, bounded, historical aggregation, and a burst alert optimizes for recency and precision. One artifact should not serve both.