PCA cheat sheet

The three pillars are metrics (aggregated numeric time series), logs (discrete timestamped events), and traces (causal request paths); exemplars link metrics to traces via trace_id.

Four golden signals: latency (separate successful vs failed), traffic, errors, saturation - the RED method uses Rate, Errors, Duration for request-driven services.

USE method for resources: Utilization, Saturation, Errors - applied per resource (CPU, memory, disk, network) rather than per service.

SLI is a measured ratio (good events / valid events); SLO is a target percentage over a window; error budget = 1 − SLO, so 99.9% over 30 days allows ~43.2 minutes of downtime.

Cardinality is the product of distinct label-value combinations per series name; a metric with a user_id label scales with user count and is the most common cause of Prometheus memory exhaustion.

Symptom-based alerting (SLO burn, error rate, latency) pages humans; cause-based alerting (CPU high, disk filling) is better routed to tickets or dashboards.

Pull-based collection makes target liveness observable: a failed scrape produces up=0, whereas push-only systems cannot distinguish a dead agent from a quiet one.

Percentiles cannot be averaged: averaging per-instance p99 values does not yield a fleet p99; aggregate with histogram_quantile over summed buckets instead.

Prometheus is pull-based: it scrapes HTTP endpoints on /metrics at scrape_interval (default 60s) and evaluates rules at evaluation_interval; both are global config values overridable per job.

The data model is a metric name plus label set; the pair (name, labels) uniquely identifies a series, and a new label value creates a new series.

Scrape config fields: job_name (required), scrape_interval, metrics_path (default /metrics), scheme, static_configs or a service-discovery block, relabel_configs, metric_relabel_configs.

relabel_configs run before the scrape and can drop targets or rewrite the address/_param* labels; metric_relabel_configs run after the scrape and drop or rename samples.

Prometheus 2.x local storage uses 2-hour blocks; retention is set with --storage.tsdb.retention.time (default 15d); remote_write ships samples to long-term stores.

Recording rules precompute expensive expressions into new series named per the level:metric:operation convention; alerting rules define expr, for, labels, and annotations.

Service discovery (kubernetes_sd_configs, consul_sd_configs, file_sd_configs) yields targets; the meta labels (_meta*) are available only during relabeling and are dropped afterward.

staleness: if a target disappears, its series are marked stale after the scrape fails; up == 0 persists for the last known target until the target is removed from discovery.

Instant vector selectors filter by label matchers: =, !=, =, !; at least one matcher must be non-empty (metric name or a label not matching the empty string).

Range vectors use a duration suffix (http_requests_total[5m]) and are only valid inside functions like rate(), increase(), or *_over_time(); they cannot be graphed directly.

rate() and increase() require at least two samples in the range and extrapolate to the window edges; the range should be at least 4x the scrape interval to avoid gaps.

irate() uses only the last two samples - it reacts fast but is noisy and unsuitable for alerting; rate() smooths over the window.

Counter functions: rate, irate, increase, resets; gauges use delta, deriv, predict_linear, and *_over_time (avg, max, min, sum, count).

Aggregation operators (sum, avg, min, max, count, topk, quantile, stddev) group by labels via by() or drop them via without(); always aggregate away high-cardinality labels before alerting.

histogram_quantile(0.9, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes the 90th percentile from bucket series; the le label must be preserved.

Binary operators match on all labels by default; use on(label) or ignoring(label) plus group_left/group_right for many-to-one joins, and vector(1) to produce a constant series.

offset (metric[5m] offset 1d) and @ (metric @ 1700000000) shift or pin the evaluation time; subqueries use [range:step] inside an outer function.

Client library metric types: Counter (monotonic, only increases), Gauge (up/down), Histogram (cumulative buckets plus _sum and _count), Summary (client-side quantiles plus _sum and _count).

Histograms are aggregatable across instances and support histogram_quantile; summaries cannot be aggregated and their quantiles are computed on the client, so histograms are preferred for fleet-wide SLOs.

Naming conventions: base unit suffix (seconds, bytes), _total for counters, _bucket/_sum/_count for histograms; labels hold dimensions, never the metric name itself.

Label values must be bounded: never use user IDs, email addresses, raw URLs, or error messages as label values - put them in logs or traces instead.

Exporters translate third-party metrics: node_exporter for host metrics, blackbox_exporter for probe-based checks (HTTP, TCP, ICMP), mysqld_exporter, and the OpenTelemetry Collector for push pipelines.

The Pushgateway exists for short-lived batch jobs that cannot be scraped; it retains the last pushed value indefinitely, so it must not be used for ordinary long-running services.

Instrumentation density: measure requests at the handler boundary and dependencies at the client boundary; avoid measuring inside tight loops, where overhead dominates.

Alerting rule fields: alert (name), expr (a PromQL expression yielding a vector), for (pending duration), keep_firing_for, labels, and annotations (summary, description, runbook_url).

Alert states are inactive → pending (expr true but for not elapsed) → firing; an alert with no for fires immediately on the first true evaluation.

Alertmanager receives alerts via the /api/v2/alerts endpoint; routing trees match on labels to select receivers, and grouping collapses related alerts into one notification.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences mute alerts for a fixed time window with a matcher set and a creator.

repeat_interval controls re-notification of a still-firing alert; group_wait delays the first notification to batch, and group_interval sets the wait between batches.

Multi-window burn-rate alerting (e.g., 14.4x over 1h and 6x over 6h for a 99.9% SLO) detects fast and slow budget consumption while limiting false pages.

Dashboard rules: one question per panel, rate() over counters rather than raw counters, a consistent time range, and units set explicitly so axes are not misread.

Avoid dashboards as the only detection path - alerts must exist independently; dashboards serve diagnosis, and each alert annotation should link to the relevant dashboard or runbook.

The three pillars are metrics (aggregated numeric time series), logs (discrete timestamped events), and traces (causal request paths); exemplars link metrics to traces via trace_id.

Four golden signals: latency (separate successful vs failed), traffic, errors, saturation - the RED method uses Rate, Errors, Duration for request-driven services.

USE method for resources: Utilization, Saturation, Errors - applied per resource (CPU, memory, disk, network) rather than per service.

SLI is a measured ratio (good events / valid events); SLO is a target percentage over a window; error budget = 1 − SLO, so 99.9% over 30 days allows ~43.2 minutes of downtime.

Cardinality is the product of distinct label-value combinations per series name; a metric with a user_id label scales with user count and is the most common cause of Prometheus memory exhaustion.

Symptom-based alerting (SLO burn, error rate, latency) pages humans; cause-based alerting (CPU high, disk filling) is better routed to tickets or dashboards.

Pull-based collection makes target liveness observable: a failed scrape produces up=0, whereas push-only systems cannot distinguish a dead agent from a quiet one.

Percentiles cannot be averaged: averaging per-instance p99 values does not yield a fleet p99; aggregate with histogram_quantile over summed buckets instead.

Prometheus is pull-based: it scrapes HTTP endpoints on /metrics at scrape_interval (default 60s) and evaluates rules at evaluation_interval; both are global config values overridable per job.

The data model is a metric name plus label set; the pair (name, labels) uniquely identifies a series, and a new label value creates a new series.

Scrape config fields: job_name (required), scrape_interval, metrics_path (default /metrics), scheme, static_configs or a service-discovery block, relabel_configs, metric_relabel_configs.

relabel_configs run before the scrape and can drop targets or rewrite the address/_param* labels; metric_relabel_configs run after the scrape and drop or rename samples.

Prometheus 2.x local storage uses 2-hour blocks; retention is set with --storage.tsdb.retention.time (default 15d); remote_write ships samples to long-term stores.

Recording rules precompute expensive expressions into new series named per the level:metric:operation convention; alerting rules define expr, for, labels, and annotations.

Service discovery (kubernetes_sd_configs, consul_sd_configs, file_sd_configs) yields targets; the meta labels (_meta*) are available only during relabeling and are dropped afterward.

Staleness: if a target disappears, its series are marked stale after the scrape fails; up == 0 persists for the last known target until the target is removed from discovery.

Instant vector selectors filter by label matchers: =, !=, =, !; at least one matcher must be non-empty (metric name or a label not matching the empty string).

Range vectors use a duration suffix (http_requests_total[5m]) and are only valid inside functions like rate(), increase(), or *_over_time(); they cannot be graphed directly.

rate() and increase() require at least two samples in the range and extrapolate to the window edges; the range should be at least 4x the scrape interval to avoid gaps.

irate() uses only the last two samples - it reacts fast but is noisy and unsuitable for alerting; rate() smooths over the window.

Counter functions: rate, irate, increase, resets; gauges use delta, deriv, predict_linear, and *_over_time (avg, max, min, sum, count).

Aggregation operators (sum, avg, min, max, count, topk, quantile, stddev) group by labels via by() or drop them via without(); always aggregate away high-cardinality labels before alerting.

histogram_quantile(0.9, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes the 90th percentile from bucket series; the le label must be preserved.

Binary operators match on all labels by default; use on(label) or ignoring(label) plus group_left/group_right for many-to-one joins, and vector(1) to produce a constant series.

offset (metric[5m] offset 1d) and @ (metric @ 1700000000) shift or pin the evaluation time; subqueries use [range:step] inside an outer function.

Client library metric types: Counter (monotonic, only increases), Gauge (up/down), Histogram (cumulative buckets plus _sum and _count), Summary (client-side quantiles plus _sum and _count).

Histograms are aggregatable across instances and support histogram_quantile; summaries cannot be aggregated and their quantiles are computed on the client, so histograms are preferred for fleet-wide SLOs.

Naming conventions: base unit suffix (seconds, bytes), _total for counters, _bucket/_sum/_count for histograms; labels hold dimensions, never the metric name itself.

Label values must be bounded: never use user IDs, email addresses, raw URLs, or error messages as label values - put them in logs or traces instead.

Exporters translate third-party metrics: node_exporter for host metrics, blackbox_exporter for probe-based checks (HTTP, TCP, ICMP), mysqld_exporter, and the OpenTelemetry Collector for push pipelines.

The Pushgateway exists for short-lived batch jobs that cannot be scraped; it retains the last pushed value indefinitely, so it must not be used for ordinary long-running services.

Instrumentation density: measure requests at the handler boundary and dependencies at the client boundary; avoid measuring inside tight loops, where overhead dominates.

Alerting rule fields: alert (name), expr (a PromQL expression yielding a vector), for (pending duration), keep_firing_for, labels, and annotations (summary, description, runbook_url).

Alert states are inactive → pending (expr true but for not elapsed) → firing; an alert with no for fires immediately on the first true evaluation.

Alertmanager receives alerts via the /api/v2/alerts endpoint; routing trees match on labels to select receivers, and grouping collapses related alerts into one notification.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences mute alerts for a fixed time window with a matcher set and a creator.

repeat_interval controls re-notification of a still-firing alert; group_wait delays the first notification to batch, and group_interval sets the wait between batches.

Multi-window burn-rate alerting (e.g., 14.4x over 1h and 6x over 6h for a 99.9% SLO) detects fast and slow budget consumption while limiting false pages.

Dashboard rules: one question per panel, rate() over counters rather than raw counters, a consistent time range, and units set explicitly so axes are not misread.

Avoid dashboards as the only detection path - alerts must exist independently; dashboards serve diagnosis, and each alert annotation should link to the relevant dashboard or runbook.