PCA: Alerting and Dashboarding

Alerting and Dashboarding

This domain covers alerting rules, the Alertmanager pipeline, and dashboard design with Grafana and PromQL. Expect questions on rule YAML, alert states, routing, grouping, inhibition, silences, and panel queries.

Core rules

• Alerting rules live in rule files with groups; each rule has alert, expr, for, labels, and annotations. Prometheus evaluates expr on evaluation_interval; a true expression moves the alert to pending, then firing after for.
• Alertmanager receives firing alerts, groups them, routes them by label matchers, inhibits lower-severity alerts, and applies silences. It does not evaluate PromQL.
• Recording rules precompute expensive queries and follow naming such as level:metric:operation, for example job:http_requests:rate5m.
• Dashboards should answer a question at a glance: use variables for job and instance, show rates not raw counters, and pair symptoms with causes. Grafana alerts are separate from Prometheus rules.

Worked example

A rule: alert: TargetDown, expr: up{job="api"} == 0, for: 5m, labels: {severity: critical}, annotations: {summary: "{{ $labels.instance }} is down"}. When up becomes 0, the alert is pending for 5 minutes, then firing; if up returns to 1 before 5 minutes, it clears without notifying. The $labels template injects the instance label. Reasoning: for suppresses flapping but delays notification by up to 5 minutes, so pair it with a fast-burn rule when needed.

Traps

Putting for on the wrong side of the YAML; expecting Alertmanager to resolve alerts Prometheus still reports; routing by annotations instead of labels; creating a silence that hides all severities; graphing raw counters so panels only rise; forgetting that a missing series does not trigger up == 0 unless the target still exists.