Log Analytics: Ingest, Processing and DQL on Grail
Log Analytics tests the path a log line takes from a file or API into Grail, what processing can change it on the way, and how you query and summarise it afterwards.
Ingest paths
- The OneAgent log module uses automatic detection rules for common paths and formats; you confirm or extend these rules per host group and technology. If the module is inactive, logs are simply absent.
- The log ingestion API accepts records from anything unmonitored, subject to token scopes and payload limits.
- OpenPipeline handles processing at ingest: parsing, masking, enrichment, routing to buckets, and dropping unwanted records.
- Timestamps matter: if no timestamp is detected, ingest time is applied, so backfilled or misformatted logs land in the wrong window and break correlation.
Processing and querying
- Dynatrace Pattern Language defines parse patterns for unstructured lines, extracting fields at ingest.
- At query time, DQL commands such as
fetch logs,parse,filterandsummarizeoperate on stored records; query-time parsing leaves storage untouched but costs compute on every run. - Masking applied at ingest is irreversible. Query-time masking protects the output, but the sensitive value remains stored.
- Buckets govern retention and access, so identical records can be invisible to a user querying the wrong bucket scope.
Worked example
An nginx access log arrives as one unstructured line. Define a DPL pattern extracting the status code and upstream time, then summarise errors over the last hour with fetch logs | parse ... | filter status >= 500 | summarize count() by upstream. Reasoning: parsing must happen before aggregation, and the field must be typed so numeric comparison works; filtering on an unparsed string silently returns nothing.
Traps that cost marks
- Field names and values are case-sensitive in filters; mismatched case yields empty results.
- Duplicate records when both the OneAgent log module and an API feed the same file.
- Assuming ingest masking can be undone by removing the rule later.
- Forgetting that log level extraction depends on a working parse pattern.