Switch failure storm and a slow instance-variable dashboard

The current group_by: [alertname] groups all 60 NodeDown alerts into one notification per alert name, but different alert names still produce separate pages. Add cluster and rack to the grouping so a rack failure becomes one notification per rack, not six per alert name. More importantly, use inhibition: a SwitchDown alert with matching cluster and rack should inhibit NodeDown alerts for nodes on that rack, because the switch failure explains them. Keep the NodeDown alert active in the UI and dashboards; inhibition only suppresses notification. Route switch alerts to the network team and node alerts to the platform on-call. Set group_wait long enough to collect the burst, for example 30-60s, and group_interval to avoid repeated pages while the incident is active. Do not inhibit on alertname alone; use equal labels such as cluster and rack. Test with amtool or a rehearsal to ensure a single node failure still pages.

up == 0 fires when Prometheus can scrape the target but the target responds with an error or times out; it does not fire when the target disappears from service discovery, because then the series becomes absent rather than 0. So the alert may miss a deleted node and may fire on a brief scrape failure. Add a for duration, such as 2-5 minutes, to suppress single-scrape blips; the exact value depends on scrape interval and how quickly the team needs to act. If the node can be removed from discovery, pair up == 0 with an absent(up{job="node",instance="..."}) rule or use a NodeDown metric from a separate exporter. Keep the alert expression simple and put the duration in for; do not encode a long duration in the PromQL unless you need a different evaluation. Test with a firewall drop and with a target removal. A switch failure will produce both scrape failures and absences, so the inhibition rule must cover both alert names.

The $instance variable has 300 values, so Grafana may issue one query per selected value or render 300 series per panel. Reduce the variable to a smaller, meaningful set: cluster, rack, or service first, then instance only after a filter. Use a query variable with a label filter and enable the Include All option with a regex, rather than rendering every instance by default. Replace the fixed 30s rate window with $__rate_interval so the range matches the scrape interval and panel step; a 30s window at a 30s scrape is too short and produces noisy, expensive queries. Add recording rules for the expensive aggregations and have panels query the recorded series. Limit panels to the signals needed for triage, and use topk rather than plotting all series. Finally, check the time range: a 7-day range with a 30s step is millions of points; use a larger step or a shorter default range.