Splunk Power User: Fields, Field Extraction, and Regular Expressions

Layers of fields

Default fields are always present. Extracted fields appear at search time from props.conf rules for the sourcetype, from automatic lookups, or from commands like rex. The field sidebar splits Selected (used in your search), Interesting (common or unusual), and All fields.

Three mechanisms get confused constantly: a field alias (same value, new name), a calculated field (an eval expression evaluated at search time), and a lookup (an external table joined on a key). Index-time extraction is rarely appropriate - search-time extraction keeps flexibility and avoids reindexing.

Regex is the core skill

Named capture groups, (?<name>pattern), are what rex and the Field Extractor turn into fields; an unnamed group produces nothing. .* is greedy and will swallow text you wanted - prefer [^ ]+ or lazy .*?. Escape special characters (\. for a literal dot); \d is a digit, \w a word character, \s whitespace, and (?i) makes a pattern case-insensitive. rex matches anywhere in the string, so anchor with ^ and $ only when you mean it. Use rex field=<otherfield> to run regex against something besides _raw, and max_match=0 to capture every match.

Field names are case-sensitive; field values are matched case-insensitively by default.

Worked example:

| rex field=_raw "(?<method>[A-Z]+) (?<uri>/\S+) HTTP/\d\.\d"

This pulls the HTTP method and URI out of a web access log; a second rex with (?<status>\d{3}) grabs the response code.

Rule of thumb: prototype with inline rex (temporary, applies only to your job); once the pattern works, save it as a field extraction for the sourcetype so every user gets it.