The Total That Missed Some Cents

The regex has no match limit and no boundary, so on a two-total= event it captures both values and amount becomes a multivalue field.

That explains every symptom at once:

• table user, amount renders comma-joined values - the visible multivalue signature.
• stats sum(amount) is not defined for multivalue input; those events contribute nothing (a null), so revenue drops while the event itself is still present.
• The panel's count differs from stats count only because count is computed off a different pipeline stage, reinforcing that the issue is field-level, not event-level.

Proof, in this order:

index=app sourcetype=app:checkout
| rex field=_raw "total=(?<amt>[^\s]+)"
| eval n=mvcount(amt) | where n>1 | table _raw n amt

Even simpler, use the UI: click the amount field in the sidebar and inspect events flagged as having multiple values.

Roughly 11% of revenue is consistent with refund volume - confirm by comparing the raw count of two-total= events against the refund line count.

Prefer a boundary-anchored regex that stops at the first token:

\btotal=(?<amount>[0-9]+(?:\.[0-9]+)?)(?=\s|$)

Key differences from the original:

• [0-9]+(?:\.[0-9]+)? cannot swallow the rest of the line if whitespace is ever missing.
• The lookahead (?=\s|$) forces the match to end at a token boundary, so total=149.99 cart_id=CA-8821 captures 149.99, not 149.99 or a trailing key.
• If a field must be single-valued no matter what, add max_match=1 in rex - but understand that this chooses a value (the first match) rather than fixing ambiguity.

Validation before promoting:

index=app sourcetype=app:checkout earliest=-7d
| rex field=_raw "\btotal=(?<amount_test>[0-9]+(?:\.[0-9]+)?)(?=\s|$)"
| eval n=mvcount(amount_test)
| stats count BY n

n must be 1 for every event, and null counts must be zero except for genuinely malformed lines (which you list and inspect). Also run | fieldsummary amount_test - min/max and null percentage should match finance's expected range. Only then commit to props.conf, remembering extraction changes do not retroactively alter already-indexed _raw (good - the fix is search-time and applies to history).

The inline rex runs later in the search pipeline, so it overwrites the props-derived amount for that search only. The panel turns green; every other consumer of amount - other dashboards, alerts, ad hoc analysis, the data model - stays wrong. You have created a permanent fork where two searches over the same sourcetype disagree on the same field name.

The deeper problem is ownership. Two regexes now claim amount: a props.conf transform and an ad hoc pipeline command. Anyone tuning the sourcetype will break a dashboard they cannot see.

Rules of thumb:

• Inline rex is for exploration and one-off diagnostics - use it to prototype the regex, then move it.
• If a correction is genuinely panel-specific, rename the field (amount_test) so no name collision exists and the difference is visible in the search.
• Field names should have exactly one owning extraction in a given app context.

After promoting the fix to props.conf, delete the inline rex and re-run the panel to confirm it still matches, and grep saved searches in the app for the field name so you catch the other consumers the inline patch hid.