Reload failure after a scrape-interval override
One Prometheus server scrapes 3,800 targets across three jobs. The global scrape_interval is 30s, but the api job overrides it with scrape_interval: 10s and scrape_timeout: 15s. After a config change, SIGHUP reload fails and the running config stays unchanged. The server holds 12 million active series, has a 64Gi memory limit, and restarts from OOM about every six hours. Retention is 30 days on a 2TB volume, and remote write to a managed Prometheus backend is 20 minutes behind. The team proposes either raising the memory limit to 128Gi or lowering retention to 15 days. They also want to know whether remote write can replace local storage for dashboards. A recent incident showed gaps on some targets during the OOM restarts.
1. Why does the reload fail, and what is the minimal correct change?
Read the worked answer
Prometheus validates the whole config on reload, so one invalid job prevents any change from taking effect. The api job sets scrape_interval: 10s and scrape_timeout: 15s. A scrape timeout longer than the interval is invalid because the next scrape would be due before the previous one is allowed to finish; Prometheus rejects it. The minimal fix is either raise the interval to at least 15s or lower the timeout to less than 10s, for example 8s. Choose based on target scrape duration: if the api targets normally take 12s, a 10s interval is not viable even with a valid timeout; move to 20s or 30s. Run promtool check config before reloading. Also check that the override is on the job, not the global, because a global timeout of 15s with a global interval of 30s would be valid but would not fix the api job. After reload, confirm the target's scrape duration and timeout errors in the targets page.
2. The server has 12 million active series and a 64Gi limit. Should the team raise memory or shard? Show the reasoning.
Read the worked answer
Estimate memory from active series first: at roughly 2-3KB per series, 12 million series is about 24-36Gi before head chunks, query buffers, and remote-write queues. A 64Gi limit is therefore tight, and OOM every six hours suggests query load and ingestion peaks matter as much as the steady state. Raising to 128Gi may stop the OOMs but does not fix the cardinality, and it increases failure blast radius. The better sequence is: (1) find the jobs and labels responsible with topk by series count; (2) drop unused labels and aggregate with recording rules; (3) if still too large, shard by cluster or job using hashmod so each Prometheus owns a subset; (4) keep a global view through remote write or federation. Lowering retention from 30 to 15 days reduces disk, not active-series memory, so it will not solve the OOM. Remote write does not replace local memory either: Prometheus still ingests and stores the WAL and head block.
3. What does the 20-minute remote-write lag tell you, and what would you check first?
Read the worked answer
Remote-write lag means samples are being queued faster than the backend accepts them, or the network or backend is rejecting or slowing requests. First check the remote-write metrics: queue capacity, current shards, samples in, samples dropped, failed samples, and the highest sent timestamp per shard. If the queue is full and samples are dropped, increase shards cautiously or reduce the number of series being sent via write_relabel_configs. If there are retries and 429 or 5xx responses, the backend is the bottleneck, not Prometheus; raising local retention will not help. Check network latency and DNS from the Prometheus pod. Also confirm the remote-write URL and auth are correct and that a recent config change did not add a huge job. A 20-minute lag with 30-day local retention still allows local dashboards, so local queries can remain served from the head block while remote write catches up. The risk is alerting from the remote backend during the lag.