Missing alerts after an ingest spike and a new tenant space

A rule only sees what is indexed when it runs. Its search window is bounded by the interval plus the additional lookback. If APAC events land 2 to 4 minutes late, their @timestamp falls outside the window on every run, so the events are never evaluated. A 55% hit rate is the signature of a partially delayed feed: on-prem events are caught, buffered ones are not.

Two candidate changes:

1. Timestamp override to event.ingested. The window is then based on index time, so late arrivals are caught on the run after they land. Cost: event.ingested is not immutable. Reindexing, restoring a snapshot, or replaying an agent queue rewrites it, which can suppress or duplicate alerts, and time-correlating rules (EQL maxspan, threshold rules) will measure index time rather than event time. Use it for the specific delayed sources, not cluster-wide.
2. Increase the additional lookback to about 10 minutes. Cost: query load grows roughly with (interval + lookback) / interval, and overlapping windows can duplicate alerts unless you add alert suppression on host.name + process.entity_id.

Before either, check the rule's execution status and duration. A run that takes longer than 5 minutes is skipped, which also produces gaps that look identical to missing data. Compare execution duration against the interval, and check agent buffering and proxy behaviour, because removing the delay is cheaper than widening every window.

Spaces scope Kibana saved objects and the alert index, not the data. Raw events live in shared data streams, so anyone whose role grants index privileges sees them in Discover regardless of space. Alerts are written to .alerts-security.alerts-<space>: a rule created in soc writes only to the soc alert index, and tenant-a's Alerts page reads only its own index. Hence events visible, alerts empty.

Also confirm the rules are genuinely enabled in tenant-a. Rule enablement is per saved object per space, so a rule imported into soc does not run in tenant-a just because the same rule ID exists there.

Options:

• Duplicate the rules into tenant-a (import with the spaces parameter, or duplicate in the UI). Correct isolation, but roughly doubles detection-engine load and alert write volume. With limited compute this may be the thing that breaks scheduling.
• Single SOC space plus filtering. Give each tenant its own data streams and role-based index privileges, or document-level security on a tenant field if the license supports it. Much cheaper; you lose the clean UI separation, so analysts must be trained on the tenant filter.
• Hybrid: keep the noisy, high-volume rules in one space and duplicate only the tenant-specific subset.

Finally, verify the rule's API key still has read on the indices it needs. A key whose owner lost privileges makes a rule fail with authorization errors that surface as zero alerts.

Change first:

• Reduce window overlap before reducing coverage. Audit every rule for an additional lookback that was widened by habit, and for intervals shorter than the data's actual arrival latency.
• Move bulk, low-query data. Endpoint network events and verbose syslog streams are candidates for faster ILM rollover to warm. Keep the last 7 days hot for the hunt workflows that actually use it.
• Increase max_signals awareness, not max_signals. If rules are capping at 100 signals per run, fix the rule filter rather than raising the cap; a capped rule produces truncated, not fewer, alerts.

Refuse to:

• Disable prebuilt rules wholesale to cut load. Instead scope rules to the data streams they need; a rule reading three index patterns instead of eleven is often 60% cheaper.
• Shorten hot retention below what investigations need. Alert triage needs maybe 14 days of hot; long-tail hunting can tolerate warm latency.
• Add Kibana/alerting throughput by scaling the whole cluster. If only scheduling is behind, add Kibana nodes dedicated to alerting, or move rule execution off the nodes serving Discover.

Measure with rule execution duration and status per rule before and after; the goal is every rule finishing comfortably inside its interval, not a lower alert count.