Stack Traces Land in the Wrong Day

A universal forwarder's job is to move bytes. Unless you deliberately configure forwarder-side line breaking, event breaking and timestamp extraction for forwarded file data happen on the indexer (in the parsing queue) or on a heavy forwarder if one is in the path.

SHOULD_LINEMERGE and TIME_PREFIX are exactly the settings that control breaking and timestamp extraction. Placed on the UF in a normal deployment, they are simply not consulted for this data path - which is why three weeks elapsed with no behavior change. The engineer changed a file that is never read for that decision.

Correct placement: in a props.conf for [orders:log] delivered to the components that parse - the indexers here, or the heavy forwarders if the architecture inserts them. On a UF, props.conf is only relevant for settings dedicated to forwarder behavior (such as explicitly enabled forwarder-side line breaking) or for transforms.conf-based filtering.

Diagnostic habit worth keeping: when a props.conf change 'does nothing,' the first question is not 'is my regex wrong' but 'does this component parse this data?' A quick check of splunk btool props list orders:log --debug on the actual parser tells you whether the setting is even in force there.

With clustering, peer configuration must come from the cluster bundle managed by the master. Do not copy props.conf onto individual peers.

Procedure:

1. Create an app under $SPLUNK_HOME/etc/master-apps/<app>/ on the master (default/ holds the props.conf).
2. Validate the bundle, then apply it (splunk validate cluster-bundle, splunk apply cluster-bundle).
3. Confirm with splunk show cluster-bundle-status that every peer reports success and no restart-required state.

Why idx-01's hand edit hurts: cluster bundles overwrite or supersede locally placed configuration in the bundle's scope, so idx-01's edit is either lost or, worse, creates a period where that peer parses differently from its neighbors. Parsing happens at index time on whichever peer accepted the data, so if idx-01 breaks and timestamps events one way while idx-02 does it another, the same sourcetype has inconsistent _time and event boundaries depending on routing luck. Buckets then disagree with each other, fixup copies the inconsistency to other peers, and searches over the index get results that vary by which peer served the bucket.

Also note the search-head side: if the search heads are in a search head cluster, search-time props belong in a shcluster app deployed with splunk apply shcluster-bundle. Index-time and search-time settings travel through different bundles.

For [orders:log]:

SHOULD_LINEMERGE = false
LINE_BREAKER   = ([\r\n]+)(?=\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\])
TIME_PREFIX    = ^\[
TIME_FORMAT    = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE       = 10000

SHOULD_LINEMERGE = false alone is not enough - without a LINE_BREAKER that anchors on the leading bracketed timestamp, Splunk still cuts at the wrong places and the continuation lines become their own events. Once lines are correctly grouped, TIME_PREFIX plus an explicit TIME_FORMAT stops Splunk from falling back to heuristic timestamp detection, which is what is producing the 3-9 hour skew (it was picking a timestamp from inside a stack trace). MAX_TIMESTAMP_LOOKAHEAD must be long enough to cover the bracketed prefix.

The existing data will not be corrected. These are index-time settings: they are applied as events are parsed, and the resulting _time and event boundaries are stored in the buckets. Three weeks of mis-parsed data stays mis-parsed, and so does the six months of data indexed before the idx-01 edit.

Options:

• Reindex the affected buckets (| delete on the bad range, then re-feed the files from the forwarders, or use a one-time batch input from an archive). Costly in license volume and time - the same bytes count again.
• Accept it and work around at search time: normalise _time from the extracted prefix with | eval _time = strptime(replace(raw_field,...), "%Y-%m-%d %H:%M:%S") in a macro or calculated field for historical reports, while new data arrives correct.

Recommendation: fix parsing via the bundle first and verify on a single host's new data before committing to a reindex - a reindex that runs before the parser is fixed just reproduces the problem at full cost.