Fourteen Correlation Searches Were Skipped Overnight

The chain is capacity contention, and the observable symptom (fewer notables) is a detection gap, not evidence of a quiet night. Verify in this order:

1. Confirm the skips are real and identify which searches skipped. Read the skipped-search messages in the job history or index=_audit and list the 14 by name and severity. If they are low-severity, the gap is smaller than feared; if the identity and endpoint brute-force rules are in the list, the shift's 12 notables may reflect a near-total blind spot.
2. Attribute the load. The six 5-minute wallboard reports with 60-minute lookbacks are 12 runs/hour each - 72 scheduled executions per hour across the cluster from a wallboard - and each holds a concurrency slot while it runs. Compare their runtimes with the correlation searches'.
3. Check the acceleration build's own consumption. Data model acceleration runs as scheduled searches on the search head. A 95-minute build instead of 20 suggests either volume growth or contention; either way the build occupied slots for 95 minutes right when the overnight detection window mattered.
4. Check for the second-order failure. Because most correlation searches use tstats summariesonly=t, a slow or incomplete acceleration build means those searches return stale or partial summaries even when they do run. A skipped search is loud; a stale summary is silent. Look at the model's build status and percent complete at 05:00, and reconcile one rule against raw data.

Only after that should scheduling change.

Priority is a function of detection value per unit of concurrency, not of who created the search.

Protect (dedicated slots, unchanged cron):

• Correlation searches with high severity and short effective windows - brute force, impossible travel, endpoint malware, privilege escalation. These are the detections whose value decays fastest if late, and they are the ones the analysts rely on.
• The acceleration build itself. It is not a report; it is the substrate that most detections read from. Starving it silently degrades dozens of rules at once.

Move:

• The six wallboard reports: raise from 5 minutes to 15-30 minutes, narrow the lookback to what the panel actually displays, and stagger their cron so they do not align with the correlation search cadence. A wallboard does not need 5-minute freshness; a detection does.
• Genuinely low-value correlation searches to off-peak or longer intervals, after confirming their lookback covers the new interval, so a rule that runs every 15 minutes still sees every event.

Separate the acceleration build:

• Move the Network_Traffic build to a quiet window and check why 20 minutes became 95 - model scope, searchable data growth, or added fields. A build that is growing 5× will keep colliding no matter how you schedule the reports.

Document the constraint: with 3 × 12 slots and 210 searches, the deployment is over-subscribed by design and scheduling will keep being the lever until capacity is addressed.

Both changes make the situation worse, for different reasons.

Concurrency to 20: each member now admits 20 simultaneous searches with the same CPU and memory. Individual runtimes lengthen, so searches occupy slots longer and overlap more; skipped-search events typically increase, and you add risk of memory pressure and unstable search heads during the overnight window. Raising a concurrency limit does not create capacity; it just changes which resource becomes the bottleneck and makes the failure mode less predictable.

24-hour lookback on skipped searches: the searches skipped at 02:00-05:00 will, on their next run, sweep 24 hours and re-detect everything they already detected in that period plus the missed window. Unless each rule throttles or deduplicates on a stable key (risk_object plus rule name, or a notable event key), that produces a duplicate-notable storm and re-scores entities in the risk index for events already counted, which distorts RBA aggregates for the next 24 hours. The correct pattern is a bounded catch-up over the exact missed window - a one-off backfill search over 02:00-05:00 - not a permanent widening of the rule's lookback.

What to say instead: the 12 notables are a known, bounded gap over a specific three-hour window; run a compensating search over that window now, record the gap in the SOC log, fix the wallboard cadence and the acceleration schedule, and treat capacity as a separate change request with measured justification. Never leave an unrecorded gap in the detection timeline.