Alert Storm During a Rack Outage

group_by: ['alertname', 'instance'] makes one group per alert name per host. Forty unreachable hosts means forty NodeDown groups, and each brand-new group waits only group_wait before flushing - so the 40 messages are the configured behaviour, not a bug.

Group by what the responder actually acts on:

group_by: ['cluster', 'alertname']   # or ['rack', 'alertname'] during a rack event
group_wait: 30s
group_interval: 2m
repeat_interval: 4h

One notification per alert name now lists all 40 hosts. The timings do different jobs:

• group_wait is the bundling window - 30 s is right for a paging receiver.
• group_interval controls how soon a changed group is re-sent; 5 m is too slow during an incident where new hosts keep appearing, so 1–2 m.
• repeat_interval controls re-notification of an unchanged group; keep it long (4 h) so a persistent alert does not re-page every five minutes.

Route by severity as well, so warnings land somewhere other than the paging channel.

inhibit_rules:
  - source_matchers:
      - alertname = "NodeDown"
    target_matchers:
      - alertname = "DiskWillFillIn4Hours"
    equal: ['instance']

Inhibition is evaluated inside Alertmanager against currently firing alerts: while a matching source alert is active, matching targets are suppressed and marked as inhibited in the UI, and they become active again when the source resolves.

Pitfalls worth naming:

• equal requires the listed label to exist on both alerts with identical values. If one rule sets instance and the other uses host, the rule silently never matches - verify in the Alertmanager UI that the target shows "inhibited by".
• Inhibition does not stop rule evaluation or reduce Prometheus load; it only silences notification delivery. The alerts still exist in the API.
• Do not inhibit page-severity alerts with a source that has a different remediation. Suppressing an unrelated page because the node is down can hide the one signal that mattered.
• A broader companion rule NodeDown inhibiting HighErrorRate on the same cluster is usually correct; over-inhibiting narrows the blast radius of what the responder can see.

The query is asking for far more work than the panel can render usefully:

• A [1h] rate window at a 15 s step means each bucket series contributes ~240 samples to every evaluation point.
• 30 days at 15 s step is roughly 172,800 steps per time series, multiplied by every le bucket across every instance.
• The step is finer than the underlying signal needs - a p99 trend over 30 days does not change every 15 s.

Changes, in order of impact:

1. Precompute with a recording rule. Evaluate every 60 s a rule such as sum by (le) (rate(http_request_duration_seconds_bucket[5m])), and let the dashboard query the recorded series. The expensive scan then happens once per minute instead of once per panel refresh, and the result is stored in the TSDB like any other metric.
2. Coarsen the panel. Set the step to $__rate_interval or at least 60 s; on long ranges, minutes are appropriate.
3. Shorten the rate window. [5m] is enough at a 15–30 s scrape interval; an hour-long window adds no statistical benefit.
4. Use downsampled or long-term storage for 30-day trends rather than raw high-resolution data, and raise the proxy timeout deliberately rather than by accident.

Record the bucket aggregation, not the quantile - histogram_quantile over pre-aggregated le series is valid; averaging pre-computed quantiles is not.