Detection Engineering: Prebuilt and Custom Rules, Alerts, and Exceptions
Rule types and when they apply
The detection engine supports several rule types, and picking the wrong one is the most common exam trap. Custom query rules match single events (KQL or Lucene). Threshold rules count matching events over the rule interval, grouped by fields such as source.ip - the right tool for brute-force patterns. EQL rules match sequences and process lineage (process where process.parent.name == "winword.exe"). Event correlation rules use EQL sequences with by join keys and optional until to express multi-stage behavior. Indicator match rules join events against a threat indicator index whose documents carry ECS threat.indicator.* fields. New terms rules alert the first time a value (up to three key fields, no arrays) appears in a lookback window. ML rules reference anomaly detection jobs, which must exist in the same space. Newer versions add ES|QL rules.
Scheduling, duplicates, and exceptions
Each rule runs on an interval and searches its interval plus an additional lookback; the engine tracks processed events to avoid duplicate alerts. Exceptions attach to a rule and suppress matching events before alert generation - they are conditions, not deletions, and can be shared via exception list containers. Value lists are separate uploadable lists, commonly used with indicator match. Alert suppression on a rule collapses repeated alerts by grouping fields while still recording the match.
Traps
Changing a rule to a type that cannot express your logic (for example using a query rule to "count" events) silently produces near-constant alerting. Prebuilt rules require installation and enablement, and their updates are reviewed separately from custom rules. Rules need read access to the target indices plus write access to the alerts index.
Worked example: "Alert when one source IP fails 10 logins in 5 minutes" → threshold rule, query event.category:authentication and event.outcome:failure, threshold 10, group by source.ip, interval 5m - not a query rule with ten conditions.