I85PREP
⌘ K
Splunk Enterprise Certified Admin study materials
Splunk Enterprise Certified Admin study resources

Two Monitors, One Host, No Data

Splunk Enterprise Certified Admin · Put your knowledge to work in context.

All case studies

Two Monitors, One Host, No Data

Sixty Linux universal forwarders are managed by a deployment server. The serverclass linux_app pushes app TA_applog (version 1.3), which contains default/inputs.conf with [monitor:///opt/app/logs/*.log], index = app, disabled = 0. Forty of the sixty hosts send nothing from that path; the other twenty are fine. On one silent host, splunk btool inputs list monitor:///opt/app/logs --debug shows the winning stanza coming from $SPLUNK_HOME/etc/system/local/inputs.conf - a file created 14 months ago by a contractor, containing disabled = 1. Nobody remembers it. Separately, etc/apps/TA_applog/local/inputs.conf exists on five hosts with index = app_preprod instead of index = app.

1. Why does the app default lose to the contractor's file? State the precedence rule and explain the five odd hosts at the same time.

Read the worked answer

Configuration files are merged by location, and for a given .conf file the precedence is, highest to lowest:

  1. $SPLUNK_HOME/etc/system/local/
  2. $SPLUNK_HOME/etc/apps/<app>/local/
  3. $SPLUNK_HOME/etc/apps/<app>/default/
  4. $SPLUNK_HOME/etc/system/default/

(etc/users/<user>/<app>/local/ sits above app-level for that specific user's search-time settings, but that does not apply to inputs.conf on a forwarder.)

So the contractor's system/local stanza wins over anything inside TA_applog, and disabled = 1 silently kills the monitor on all forty hosts that received that file at build time. The twenty working hosts simply never had the file.

The same rule explains the five odd hosts: within the app, local/ beats default/, so their index = app_preprod overrides the app's default value. That is not a mystery either - it is the same precedence ladder one level down, and it means those five hosts are quietly writing to the wrong index (a data-governance problem even though data is flowing).

Key takeaway: etc/system/local is the highest-priority location on the host and is invisible to the deployment server, so it is the most common source of 'the app is deployed but nothing happens' drift.

2. How would you prove which file actually wins on a given host without guessing, and what are the traps in using that tool?

Read the worked answer

Use btool on the affected host:

$SPLUNK_HOME/bin/splunk btool inputs list monitor:///opt/app/logs --debug
  • Without --debug, btool prints the merged, effective configuration - i.e. what splunkd will actually use. That answers 'what is in force' but not 'why'.
  • With --debug, btool prints every file it read, in load order, followed by the merged result, so you can see the later/higher-precedence file supplying the value that survives. The stanza that appears last for a setting is the winner.

Traps:

  1. Running btool on the deployment server proves nothing. The DS reports its own config, not the client's. On clustered indexers, btool on a peer also does not show bundle-sourced settings the way you might expect.
  2. btool applies file precedence, not runtime state. A stanza can be correct and still not be active because splunkd has not reloaded.
  3. Narrow the output with the --app=<app> flag when you are chasing a specific app.

After fixing the file, push a restart via the deployment app (restartSplunkd.conf) and re-run the btool command plus a splunk list monitor check on the host to confirm the monitor is live.

3. What is the durable fix so this cannot recur across a 60-host (and growing) fleet?

Read the worked answer

Remove the drift, then remove the ability to create it.

  1. Delete or neutralize the system/local/inputs.conf stanza on every affected host - it is not deployable content and cannot be managed by the DS, so it will resurrect itself with every clone/image if left in place. Grep the fleet for etc/system/local in disposable config files.
  2. Decide the real index value for the five hosts. If app_preprod is legitimate, move it into a proper app or a separate serverclass for those hosts so it is visible and versioned, instead of a hand-edited local file.
  3. Reserve etc/system/local for host identity only - server.conf, deploymentclient.conf, outputs.conf and similar per-host settings. Publish a one-line rule to the team and enforce it in the build image.
  4. Detect drift continuously. Run a scripted input or a periodic scripted check that runs splunk btool inputs list --debug and reports when etc/system/local supplies a winning stanza for a data input; alert on non-zero.
  5. Verify the outcome, not the deployment. The deployment server showing 'deployment successful' only means files landed. Confirm with a search that counts events by host for the last 24 hours and compares the host list against the serverclass membership - the forty silent hosts should have shown up there long before the contractor's file was found.