An anomaly detection job that is too noisy to be useful

Model memory is driven by how many entities get their own statistical model, not by data volume. Memory is allocated per partition_field_value (and per detector), times the models needed for each detector function. Here it is 40 services × 1 detector - that part is fine. The damage comes from having url.path as an influencer with 30,000 distinct values plus 1,200 hosts: influencers are tracked in each bucket's model, and a very high-cardinality influencer bloats every model plus the per-bucket aggregation state.

Order of operations:

1. Fix the configuration first. Raise the memory limit only after the model count is sane; otherwise you are buying time on a design that still produces noise.
2. Drop url.path as an influencer. 30,000 values over a 5-minute bucket is mostly singletons; rare-seen values generate spurious anomalies by construction. If path-level behaviour matters, run a separate job keyed by a partition_field over a filtered set of paths (e.g. a query_delay-safe filter or an ingest-time path_group keyword with a few dozen values).
3. Keep service.name as the partition_field (it already is), and demote it from the influencer list - a partition field does not need to also be an influencer, and duplicating it inflates state.
4. Drop host.name as an influencer unless hosts are the unit of interest; 1,200 values in every model is expensive for marginal signal.
5. Then check GET _ml/anomaly_detectors/<job>/_stats for model_size_stats and memory status; set model_memory_limit from observed peak_model_bytes with headroom, not from a round number.

Finally, note that a 5-minute bucket over 150M docs/day is cheap for search but yields 288 buckets/day - modelling daily seasonality in response_time_ms needs far more than 48 hours of history before it can distinguish "normal Sunday" from "anomaly".

A single high_mean detector on response time will smear a 20-minute event across 5-minute buckets and, worse, cannot tell "traffic stopped" from "everything is fine", because a throughput drop does not raise the mean response time.

Design:

• Two detectors, not one. Keep high_mean(response_time_ms) for latency, and add low_count (or low_non_zero_count if empty buckets should be ignored) on the event rate so the outage is caught as a drop in volume. A low_non_zero_count avoids treating a quiet-but-alive service as an outage.
• bucket_span. At 5m a 20-minute outage spans 4 buckets - detectable, but the model needs the bucket to be long enough that normal traffic produces a stable count. 5m is defensible for 150M docs/day; do not go below it just to sharpen the outage edge.
• detector_description should state the unit and direction so operators read it correctly.
• Daily seasonality requires the job to have seen multiple full weeks. Until daily/weekly components are populated (analysis_config.detectors[].detector_description plus _stats model_size_stats), the job will flag the ordinary daily peak.

What it will wrongly flag:

• The Sunday outage window itself creates a step change at both edges; expect an anomaly at the recovery bucket too, since the model must re-learn after a gap.
• Services with sparse traffic will produce low_count anomalies continuously - this is the same cardinality trap as url.path. Only enable low_count for services whose baseline count is comfortably above the model's minimum, or partition out the sparse ones.
• Any deliberate deploy that shifts latency upward by a small, sustained amount will be flagged once and then absorbed - that is correct behaviour, but the team should be told so they do not chase it.

"No data" / shifted buckets almost always come from the gap between event time and index time:

• Check datafeed_config.query_delay. If documents arrive later than the job assumes, the datafeed looks at a window with no documents yet, logs No datafeed data received, and later re-processes. Set query_delay from observed ingest lag (p99 of index_time - @timestamp), not from a guess.
• Check frequency and chunking_config. Default chunking is fine at 150M docs/day; a mode: manual chunk size that is too small produces many small searches and can time out.
• Check that the datafeed's index pattern and the analyzer's time_field point at the same field and that it is a date mapped as an epoch/ISO value. A keyword-typed timestamp silently yields no buckets.
• Confirm the cluster timezone assumptions are irrelevant - ML bucketing is on UTC epoch, so a dashboard in Europe/Berlin will look shifted if it re-buckets by local day. Verify by comparing bucket.timestamp (epoch ms) with the raw documents.

Validation sequence before production:

1. Run with "analysis_limits": {"model_memory_limit": ...} set from a historical replay over 2–4 weeks, not live tail only: datafeed_config with start/end on a task, wait_for_completion.
2. Inspect _stats: model_size_stats.model_bytes vs limit, memory_status, and data_counts.invalid_dates_count / skipped_empty_buckets.
3. Review _results/records sorted by record_score and confirm the top records correspond to events operations actually recognises. If the top 50 records are all rare-path noise, the config is still wrong - fix the influencers rather than lowering anomaly_score_threshold.
4. Establish a baseline period (delete/re-create the job) so the model has not been contaminated by the noisy 48 hours, then promote it and set a custom_rules/job_id alert on record_score > threshold for the on-call channel.
5. Only then raise model_memory_limit if the replay shows genuine peak usage near the cap.