Splunk Enterprise Certified Admin: A Debug Log Flood Eats the 30-Day License Window

A Debug Log Flood Eats the 30-Day License Window

A single-license-stack deployment holds a 500 GB/day Enterprise license and normally indexes a steady 380 GB/day across four indexes. Nine days ago the application team enabled DEBUG logging in a new web tier; those events arrive in index web_debug and add about 1.2 TB per day on top of the baseline. Splunk Web now shows a license warning banner, and the platform owner asks whether search will be disabled and how long they have. The only remediation proposed so far is creating a 50 GB/day license pool for web_debug and moving the index into it 'so the violation goes away.' All data arrives through 400 universal forwarders; the indexers are already running at 80% CPU.

1. Is the stack currently in violation, and will search be disabled? Show the arithmetic and describe the mechanism.

Read the worked answer

Splunk enforces the license over a rolling 30-day window, not per calendar day. Quota for the window = 500 GB × 30 = 15,000 GB.

Usage in the window:

21 normal days × 380 GB = 7,980 GB
9 flood days × (380 + 1,200) GB = 9 × 1,580 = 14,220 GB
Total ≈ 22,200 GB, about 148% of quota

So yes - the stack is in violation, not merely near the limit.

Mechanism: exceeding the daily quota generates warnings in the UI banner and in splunkd.log, but the escalating consequence comes from sustained violation. If the rolling window stays over quota for a continuous period (30 days of continuous violation), the license manager disables search: search heads refuse searches for non-privileged users, and only a narrow set of licensing/administrative activity continues. The counter is not a one-way switch - when window usage falls back under quota, search is restored - but the clock does not pause for weekends or holidays.

Because the flood started nine days ago, they have roughly 21 days before search is disabled unless window usage drops. Verify with index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart sum(GB) span=1d and the Licensing usage report in Splunk Web.

2. Evaluate the proposal to create a 50 GB/day pool for `web_debug`. Will it remove the violation?

Read the worked answer

No. Pools partition the stack quota; they do not add to it.

A license pool is a slice of the stack's total. Creating a 50 GB/day pool for web_debug means that index is expected to consume at most 50 GB/day of the stack's 500 GB/day.
Exceeding a pool quota produces a warning only - it is not itself a stack violation. That is exactly why the proposal sounds appealing: the word 'violation' would disappear from the pool's context.
But the 1.2 TB/day still arrives at the indexers and is still metered against the stack quota. The rolling 30-day stack usage stays at ~22,200 GB, so the stack violation continues and search is still on track to be disabled.

Pools are a chargeback and fair-sharing mechanism, not a filtering or throttling mechanism. In fact this proposal is actively harmful: it creates a false sense that the problem is handled, and it splits accountability so the owning team sees only a pool warning while the platform team absorbs the stack violation.

A pool is appropriate only after the volume is under control - for example, to cap chargeback for that team at 50 GB/day and alert them when they exceed it.

3. What is the fastest fix that requires no new hardware, and why is filtering on the forwarders materially better than filtering on the indexers?

Read the worked answer

Attack the source first: get the application team to turn DEBUG off, or restrict it to a short window with rotation. Everything else is a stopgap.

Then filter as far upstream as possible:

At the universal forwarder - inputs.conf blacklists for whole files (e.g. blacklist = debug.* in the monitor stanza), or a forwarder-side props.conf/transforms.conf pair that routes matching events to the null queue (REGEX = DEBUG, DEST_KEY = queue, FORMAT = nullQueue).
At the indexer - the same null-queue transform, which is far weaker.

The difference is metering. License volume is counted when data arrives at the indexer. A null-queue filter on the indexer still consumes license and still costs parsing CPU on machines that are already at 80%. Events discarded on the forwarder never cross the network and are never metered, so the rolling window actually falls - which is what resets the violation clock.

Given the CPU headroom problem, forwarder-side filtering also protects search performance, and it scales: 400 forwarders each dropping their own debug lines is cheaper than four indexers parsing and discarding 1.2 TB/day.

Sequence: enable the forwarder filter today, confirm web_debug volume drops in the usage report, then re-evaluate the pool decision once the stack is back under quota.