Alerting & Dashboarding
Alerting & Dashboarding tests the path from a PromQL expression to a useful page or ticket, and from raw series to a dashboard that survives real traffic. Know rule evaluation, Alertmanager behavior, and panel query hygiene.
Alerting rules and Alertmanager
Prometheus evaluates alerting rules at evaluation_interval. An alert is pending while for has not elapsed, then firing; keep_firing_for can extend firing. Labels identify the alert and route it; annotations hold templated human text such as {{ $labels.instance }}. Alertmanager deduplicates, groups, routes, inhibits, and silences alerts. group_wait, group_interval, and repeat_interval control notification timing. Inhibition suppresses lower-severity alerts when a related higher-severity alert fires. Silences are time-bounded matchers.
Worked example
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "Error ratio above 5%"
The for: 10m prevents paging on a single bad scrape. The expression already aggregates to one vector, so the alert produces one instance rather than one per status code. A dashboard panel should use the same ratio but display it with a percentage unit and a time range longer than the SLO window.
Traps
- Alerting on a raw counter or on
irate()for a slow-burn condition. - Omitting
for, causing flapping pages. - Using high-cardinality template variables that produce hundreds of queries.
- Confusing inhibition with silencing; silences are manual and expiring, inhibition is rule-based.
- Building recording rules that change metric semantics without documenting them.