Per-core and per-container metrics overload a 450-host Agent fleet
An SRE team monitors 450 Linux hosts with Fleet-managed Elastic Agent 8.x. Each host runs the System integration, the Docker integration and a custom Prometheus scrape integration. After a capacity review, the team enables per-core CPU metrics and per-container network metrics to investigate a latency complaint. Document volume rises from 1.4 million to 11 million per day, and the metrics hot tier grows from 90 GB to 1.1 TB per week. Within two days, 38 agents show DEGRADED with the message output blocked, while the rest remain Healthy but their metrics arrive 15-25 minutes late. The Elasticsearch hot tier sits at 86 percent disk usage. ILM has a single hot phase with rollover at 30 days, no downsampling and delete at 90 days. Dashboards query max, average and 99th percentile over 30 days. The team wants to keep 90 days at lower cost and still detect a single CPU core pegged at 100 percent for 10 minutes.
1. Why are some agents DEGRADED while others are only delayed, and what would you inspect to confirm backpressure rather than an agent bug?
Read the worked answer
DEGRADED is a unit-state signal from Elastic Agent, not a measure of network quality. When the Elasticsearch output cannot accept events, the agent retries with backoff; agents that hit repeated failures mark the output unit as failed and report DEGRADED, while agents whose retries eventually succeed stay Healthy but their event timestamps fall behind. The 15-25 minute lag is the queue draining, not clock skew.
Confirm on one affected host with elastic-agent status and the agent logs under the installation data directory; look for bulk rejections, HTTP 429 and retry messages. On the cluster side check GET _cluster/health, GET _cat/thread_pool/write?v&h=node_name,active,queue,rejected, and GET _cat/allocation?v. Search Elasticsearch logs for es_rejected_execution_exception and indexing pressure rejections. The 86 percent disk level is above the low watermark, so new shard allocation is blocked, which also makes rollover-created shards harder to place. High-cardinality metrics increase bulk size and queue pressure, so the two symptoms share one cause. Fix volume first; raising queue sizes only postpones the failure.
2. Which changes reduce volume the most while preserving the ability to detect one core at 100 percent for 10 minutes?
Read the worked answer
Per-core metrics are the dominant multiplier: 450 hosts times tens of cores times a 10-second interval produces far more documents than the host-level CPU metricset. Keep per-core collection only where it answers the question. If every host needs it, raise the collection interval for the core metricset to 60 seconds, which still gives ten samples in a ten-minute window, and drop fields nobody queries with a drop_fields processor in the integration policy. For containers, aggregate network metrics per container rather than per interface, and disable metricsets for stopped containers.
Split policies: a critical-hosts policy with per-core metrics at 60 seconds, and a default policy with host-level CPU only. That removes most of the volume without losing the detection capability where it matters. Keep collection at 60 seconds rather than 10; alerting on a 10-minute sustained condition does not need sub-minute resolution.
Also check the Prometheus integration for high-cardinality labels such as request IDs or pod UIDs and drop or hash them. Cardinality, not document count alone, drives memory and disk.
3. Design the 90-day storage layout and explain how downsampling affects the ten-minute detection requirement.
Read the worked answer
Use a time series data stream for metrics so dimensions and metrics are stored efficiently, and confirm the integration writes to a TSDS-compatible data stream with host.name and core identity as dimensions. ILM: hot for 7 days with rollover on shard size or age, warm at day 8 with a 1-hour downsample, cold at day 30 with a 1-day downsample and searchable snapshots, delete at day 90.
Downsampling preserves aggregatable values such as min, max, sum and count, so a one-hour bucket still contains the true maximum CPU percentage for that hour. It does not preserve the ten-minute duration. A core pegged at 100 percent for ten minutes will appear as a high maximum in the hourly bucket, but you cannot prove from downsampled data alone how long it stayed there.
Therefore alert on raw 60-second data and keep raw metrics for at least 14 days, which covers incident review. Use downsampled tiers for capacity trends and month-over-month dashboards, and state that limitation in the runbook. Do not drop dimension fields when trimming, or the TSDS mappings and aggregations break.