Basic Searching and Search Under the Hood
What actually happens
When you run a search, the search head parses your string into search terms, dispatches to indexers, and each indexer scans time-ordered buckets using bloom filters and the tsidx term dictionary to find candidate events before any command runs. Because most fields are extracted at search time, only default fields exist when the search begins: host, source, sourcetype, index, _time, _raw, linecount, punct, and splunk_server. Everything else appears only after extraction, or because a command such as rex created it.
Search modes control extraction effort:
- Fast - field discovery off; quickest, but many fields are missing from the sidebar.
- Smart (default) - extracts the fields the search actually needs.
- Verbose - extracts every field for every returned event.
Traps that show up in questions
AND is implicit, but AND/OR/NOT must be uppercase (or is treated as a keyword). NOT and != also return events where the field is absent; add field=* when the field must exist. Double quotes force exact-phrase matching and suppress wildcards, so "*error*" matches far less than *error*. Trailing wildcards (foo*) are cheap; leading wildcards (*foo) force a broad scan.
Worked example:
index=web sourcetype=access_combined status>=500 host=* NOT host=dev-* earliest=-4h
Index and sourcetype cut volume first, the time range bounds which buckets are read, the numeric comparison uses an extracted field, and host=* prevents the NOT clause from silently dropping events with no host.
Rule of thumb: filter with index/sourcetype/time first, keywords second, field comparisons third - each earlier step removes work for every later step. Search results live in a job that eventually expires, so use Save As if you need persistence, and the Job Inspector when a search is slow.