Alert routing collapse during a 1,200-notification incident
Grafana 11 OSS with built-in Grafana Alerting manages 350 alert rules across six teams. All rules currently use the default notification policy, which sends to a single Slack contact point, #alerts-all. During a 30-minute incident, 1,200 notifications hit the channel; responders muted it. The database team's page for replication lag arrived 12 minutes late, a disk-space warning repeated every 30 seconds, staging alerts woke on-call at 3 a.m., and a planned maintenance window produced 80 alerts. Rules already carry labels team, severity, env, and service, but no notification policies use them. Contact points exist for per-team Slack channels, PagerDuty for critical, and email for low severity. The incident review asks for routing that pages the right person without flooding everyone else.
1. Diagnose the noise and the delayed page. Which alerting controls are missing or misconfigured?
Read the worked answer
All 350 rules inherit the default policy and a single Slack contact point, so routing ignores the labels that already exist (team, severity, env, service). That produces a firehose: staging, low, and critical all land in #alerts-all, and responders mute the channel, which also hides the database page. The 12-minute delay is consistent with a single contact point and Slack rate limiting plus grouping that batches unrelated alerts; critical alerts are not sent to PagerDuty at all.
The 30-second repeat suggests the policy or rule has group_interval or repeat_interval set far too short, or many alert instances are being generated without grouping by a stable label such as service. The 3 a.m. staging wake-up shows there is no env=staging route or mute timing. The maintenance window produced 80 alerts because no silence or mute timing matched the maintenance labels. None of this is a Slack problem; it is an unconfigured notification policy tree.
2. Design a notification policy tree and timing settings. Give concrete matchers and explain ordering.
Read the worked answer
Root policy: default contact point = low-noise email, group_by: ['alertname','service'], group_wait: 30s, group_interval: 5m, repeat_interval: 4h. First child: env=staging to Slack #alerts-staging, with a mute timing for weeknights 20:00-08:00 and weekends, unless severity=critical; put the critical staging exception above the mute via a child policy if needed.
Second child: env=prod, severity=critical to PagerDuty, repeat_interval: 1h, continue: false. Third child: env=prod, team=db to Slack #db-alerts, with group_by: ['service'] to keep replication-lag alerts together. Fourth child: env=prod, severity=low to email digest. Order matters: Grafana evaluates top-down and stops at the first match unless continue is true, so put specific critical routes before broad team routes. Use mute timings for recurring maintenance and silences only for one-off windows; silences expire and are easy to forget.
3. How would you roll this out without causing a new incident, and how would you validate routing?
Read the worked answer
Roll out in stages. First, inventory labels and confirm every rule has team, severity, and env; fix unlabeled rules before routing, because they will fall through to the default. Second, create contact points and test each one from the UI; for PagerDuty, send a test event to a low-urgency service. Third, apply the policy tree to a canary subset by adding a temporary label route or by testing with synthetic alert rules that carry the real labels. Create a synthetic rule per route with a short for and a known label set, and observe which contact point receives it. Fourth, check grouping behavior by firing several instances with the same service and confirming one grouped notification, not 20.
Fifth, announce the new routing, keep the old Slack channel as a temporary mirror via continue for one week, then remove it. Finally, document silence ownership and review mute timings before the next maintenance window.