Log Management
Log Management
This domain covers getting logs into New Relic, shaping them so they are queryable, and controlling cost and privacy.
Core rules
- Logs arrive via APM agent log forwarding (application_logging.forwarding enabled), infrastructure agent log forwarding (file paths in config), open-source shippers (Fluent Bit, Fluentd, Logstash, Vector, syslog-ng), cloud services (AWS Firehose, CloudWatch subscriptions, Azure Event Hub, GCP Pub/Sub), or the HTTP log API.
- Every record is stored as the Log event type with attributes such as message, level, timestamp, host, and, when correlated, entity.guid, trace.id, and span.id.
- Parsing rules use grok-style patterns with named captures to turn an unstructured line into attributes you can facet and alert on.
- Log-to-metrics rules extract counts or numeric values into Metric data, which is cheaper to retain long term and faster to alert on than raw logs.
- Drop filters remove matching records at ingest; obfuscation rules mask or hash sensitive patterns such as emails and card numbers before storage.
- Live tail streams logs in real time; NRQL queries stored data afterwards.
Worked example
An nginx access line reads: 10.1.2.3 - - [12/Mar/2025:10:00:01] "GET /checkout HTTP/1.1" 502 512. A parsing rule with named captures for client IP, method, path, status, and bytes turns status into a numeric attribute; a log-to-metrics rule counting requests where status is 500 or above then feeds an alert condition without storing every line indefinitely.
Traps
- Timestamps that are not ISO 8601 or epoch milliseconds, which breaks event ordering.
- Assuming parsing applies retroactively; rules act on records ingested after they are enabled.
- Writing drop filters that remove records another team depends on; filters are account-wide.
- Expecting log-to-metrics to keep every original attribute; only specified dimensions survive.