Elastic Certified SIEM Analyst study materials
Elastic Certified SIEM Analyst study resources

1,400 alerts/day: tuning two rules without going blind

Elastic Certified SIEM Analyst · Put your knowledge to work in context.

All case studies

1,400 alerts/day: tuning two rules without going blind

Fourteen custom rules plus prebuilt rules generate about 1,400 alerts on a weekday. Roughly 1,290 come from Potential Credential Access via LSASS Memory Dump and Suspicious Rundll32 Network Activity. Investigation shows three causes: the service account svc_veeam triggers the LSASS rule on 60 file servers every four hours because Veeam reads lsass.exe memory; the Elastic Defend agent's own elastic-endpoint.exe triggers the same rule; and an internal IT deployment tool C:\IT\deploy\agent.exe triggers the rundll32 rule on 800 workstations. Red team exercises run quarterly and use the same svc_veeam account, launched from a jump host. Compliance requires LSASS access detection to stay enabled on every server. You have three analysts and no SOAR.

1. Write the exception entries you would create and explain the combination semantics that make them safe.

Read the worked answer

Semantics first, because they determine safety. Within a single exception item, multiple entries are ANDed. Across items, the items are ORed. One item containing user.name AND process.executable AND host.name hides only that exact tuple. Three separate items would each hide their condition cluster-wide, which is the classic way to blind a rule.

Proposed items:

  • A (LSASS rule): user.name match_any: [svc_veeam] AND process.executable match_any: <Veeam binary list> AND host.name match_any: <value list 'veeam-file-servers'>.
  • B (LSASS rule): process.executable match_any: [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe] AND process.parent.executable match_any: [<agent service host>]. The EDR touching LSASS is expected behaviour; bind it to the agent's own binary so an attacker masquerading as the agent name still alerts on the parent mismatch.
  • C (rundll32 rule): process.executable match_any: [C:\IT\deploy\agent.exe] AND host.name match_any: <workstation value list>.

Use match_any with value lists rather than match with wildcards. Value lists are reviewable, live in version control, and can be attached to several rules through a shared exception list. Build one shared list for the Veeam fleet and attach it to the LSASS, credential-dump and process-injection rules rather than maintaining three copies.

2. What must not go into these exceptions, and how do you prove the change did not break red-team visibility?

Read the worked answer

What not to do:

  • Do not except user.name: svc_veeam on its own. The red team uses that account, and a bare account exception removes the account from every LSASS, dump and credential-access rule permanently.
  • Do not use host wildcards such as *file* or *srv*. They silently cover hosts added later and cannot be audited.
  • Do not disable the rule on the 60 servers, even though compliance aside it looks tempting: the whole point is that LSASS access on a server is the signal.

Proving it:

  1. Run rule preview over a window containing both benign Veeam activity and recorded red-team activity, first without and then with the exceptions, and compare alert counts and the surviving documents.
  2. Recognise what preview does not prove: it runs an ad-hoc window through the rule's query. It does not model scheduled execution with delayed indices, it does not exercise alert suppression, and it does not reproduce the exact window boundaries of a 5-minute run. It is strong evidence, not proof.
  3. After enabling, watch the rule's execution status and alert count per day for 48 hours. A drop to exactly zero on a server fleet is as suspicious as no drop at all.
  4. Note the residual gap explicitly: while the exception is placeable from the jump host, the tuple (svc_veeam + Veeam binary + fleet host list) excludes it, so a red-team run from the jump host still fires, but a run that mimics the Veeam binary path on a fleet server would not. That residual risk belongs in the exception's description field.

3. After the exceptions, the rundll32 rule still produces about 300 alerts/day from the same hosts. Exceptions or suppression, and why?

Read the worked answer

It depends on whether the residual activity is benign or merely repetitive.

  • If it is genuinely benign (same tool, same behaviour, same path), use another exception scoped as tightly as item C. Exceptions stop generation entirely.
  • If it is suspicious but low-novelty (same host doing the same thing, worth one look per shift), use alert suppression on host.name plus process.parent.entity_id with an 8-hour window. You get one alert per host per shift and a counter for the rest.

The difference matters operationally: an exception prevents the alert existing at all; suppression evaluates, writes one, and discards the remainder. Suppressed alerts still cost the full query, so suppression is a triage fix, not a performance fix, and it must not be used to paper over load problems. Exceptions usually do reduce load because the rule's query can stop matching those documents, though only if the exception is expressible in the rule's filter.

Also confirm suppressed counts are visible in the rule details. If they are not, growth in the underlying activity becomes invisible, which is exactly the pattern you need to catch before it becomes an incident. If suppression volume climbs week over week, convert the suppression back into an exception to force a re-review.