Log monitoring and log analysis
Domain 2: Log monitoring and log analysis
This domain tests turning raw log lines into searchable, alertable evidence: parsing with ingest pipelines, querying in Kibana, aggregating for dashboards, and creating rules that fire on log conditions.
Parsing and normalization
Integration pipelines already parse common sources (nginx, Apache, system logs, Windows events). For custom logs, choose grok when fields have variable structure and regular expressions are needed, or dissect for fixed delimiters, which is faster and fails less ambiguously. Multiline logs must be joined before indexing - Elastic Agent and Filebeat use a multiline configuration with pattern and negate; Logstash uses the multiline codec or filter. Normalize to ECS so searches and prebuilt dashboards work: log.level, message, event.dataset, host.name, service.name, source.ip.
Querying and aggregating
Kibana's KQL uses field: value, and/or/not, wildcards, ranges (status >= 500), and * existence. Full-text searches hit analyzed text fields; exact filters and aggregations need keyword fields. In Discover, add columns, filter by time range, and save searches. Visualizations and Lens aggregate by keyword fields; a text field without a keyword subfield cannot be used for terms aggregations. ES|QL is available in newer versions for piped transformations.
Alerting and detection
Kibana rules include logs threshold, Elasticsearch query, index threshold, and anomaly detection jobs. A logs threshold rule counts documents matching a KQL query over a window and triggers when the count crosses a threshold. Alert connectors send to Slack, email, PagerDuty, or webhooks. Rule previews and alert summaries help tune noise.
Worked example
An nginx access log line 10.0.0.5 - - [04/Jan/2025:10:15:00 +0000] "GET /checkout 500 1234 must be parsed and alerted when 5xx responses exceed 10 in five minutes. Reasoning: a grok pattern captures source.ip, timestamp, method, URL, status, and bytes; a date processor parses the timestamp; status is converted to long and url kept as keyword. A logs threshold rule uses KQL http.response.status_code >= 500 with a five-minute window and threshold above 10.
Traps
Greedy grok patterns can swallow later fields; anchor and test patterns. dissect fails if delimiters change. KQL wildcards at the start of a term are expensive. Aggregating on text returns an error or empty results. A multiline pattern that is too broad joins unrelated events. Time zone mistakes shift the window and hide events.