I85PREP
⌘ K
Splunk Core Certified User study materials
Splunk Core Certified User study resources

License Violation After a New Firewall Feed

Splunk Core Certified User · Put your knowledge to work in context.

All case studies

License Violation After a New Firewall Feed

Northwind Retail runs one Splunk indexer and one search head. Forty universal forwarders send application and system logs, about 55 GB per day in total, against a 50 GB per day license. Last month the network team pointed a new firewall feed at the same indexer, adding roughly 25 GB per day. Since then, searches that used to finish in seconds take minutes, Splunk Web shows a license violation banner, and the platform team has been asked to 'add more disk.' The security team wants firewall data kept for 90 days, while application logs only need 30 days. Nobody has touched the forwarder configurations in over a year.

1. The CIO asks whether buying more disk will clear the license violation. Where is the license actually measured, and what really happens when daily indexed volume exceeds the limit?

Read the worked answer

More disk will not help. Splunk licensing is a daily indexed-volume license (GB/day), not a storage license. It is measured on how much data is written into the index each day, tracked by the license master (the indexer itself in a small deployment). Disk capacity only affects retention - how long data stays searchable before rolling to frozen.

What happens on overage, in order:

  1. Warning - the banner and license usage warnings appear in Splunk Web and messages are sent to admins. Nothing is blocked yet.
  2. Persistent violation - if usage keeps exceeding the limit across the warning window, the deployment enters violation and search is restricted (in the extreme, non-admin users lose search).

So the CIO's question is really about volume, not capacity: 80 GB/day against a 50 GB/day license is a permanent violation no matter how much disk is attached.

2. Recommend the changes that bring indexed volume back under the license while still keeping 90 days of firewall data, and say which component should do the filtering and why.

Read the worked answer

Three levers, in order of preference:

  • Filter before indexing. Put a heavy forwarder in front of the firewall feed. Universal forwarders cannot drop events - they do not parse payload data, they only add metadata - so the filtering has to happen on a heavy forwarder (or at the indexer, but then you have already paid the license cost). Use props.conf/transforms.conf with nullQueue for noisy deny/allow chatter, or route only relevant traffic types to the indexer. That is the real fix: the 25 GB/day stops being indexed at all.
  • Reduce at the source. Ask the network team what is actually needed for investigations; a full firewall log is usually a fraction of what is useful.
  • Separate retention by index. Create a firewall index with frozenTimePeriodInSecs sized for 90 days and an app index sized for 30 days. Retention is an index setting - it does not require extra license or a bigger disk, just enough storage for the firewall index.

Net effect: indexed volume drops under 50 GB/day, and 90-day firewall retention is a retention setting rather than a licensing one.

3. The 40 forwarders need a new outputs.conf to point at the heavy forwarder. How should that configuration be delivered, over which port, and what would you check if one forwarder never picks it up?

Read the worked answer

Use the deployment server role (it can be a separate instance or co-resident on the search head in small environments). You author an app in $SPLUNK_HOME/etc/deployment-apps/ containing outputs.conf, then assign it to a server class covering the 40 forwarders.

  • The forwarder gets deploymentclient.conf pointing at the deployment server on port 8089 (the management port).
  • The forwarder checks in on its phoneHomeIntervalInSecs (default 60s), downloads the app bundle, and applies it - normally a restart of splunkd is needed unless you are using a config that the forwarder reloads.

If one forwarder never gets the config, check: (1) does its deploymentclient.conf have the right host/port and is 8089 open; (2) is the host in a server class that includes the app (whitelist/blacklist filters); (3) splunkd.log on the forwarder for deployment client errors; (4) whether someone edited outputs.conf locally, which the deployment server will overwrite or which may mask the deployed app.

Note the deployment server is a configuration mechanism, not a data path - it never touches the events themselves.