Log Management
Pipelines and processors
Logs flow: ingest → processing pipelines → indexes → archives. Pipelines are ordered and contain processors: Grok parser, date rematcher, status rematcher, service remapper, category processor, lookup processor, arithmetic processor, and attribute remapper. Reserved attributes (host, service, source, status, message, timestamp) drive facets and correlation; everything else is a custom attribute under @.
Indexes, exclusion filters, and cost
Indexes are where logs become searchable and billable. Each index has retention (3, 7, 15, or 30 days) and exclusion filters - a log matched by an exclusion filter is dropped from the index but still counts as ingested. Order matters: indexes are evaluated and the first matching rule applies. Archives (S3, GCS, Azure) keep logs long-term and rehydration pulls them back for a fee. Live Tail streams logs regardless of indexing.
Log-based metrics
Log-based metrics are generated from all ingested logs, including those excluded from indexes - a standard trap. That makes them the cheap way to alert on high-volume events you never want to index.
Sensitive data
Sensitive Data Scanner uses scanning rules with redaction or hashing to mask PII before indexing.
Worked example: a raw log line 2024-05-01T12:00:00Z WARN svc=pay msg="card declined" is parsed with a Grok rule extracting svc and msg, a date rematcher setting the timestamp, a status rematcher mapping WARN to warn, and a service remapper pointing to pay - after which @svc is a usable facet and service:pay aligns with APM.