Prometheus Certified Associate flashcards

34 cards. Read the term and the definition here. The deck at the top of this page is optional.

Four Golden Signals

Google SRE's four key metrics for user-facing systems: latency, traffic, errors, and saturation. Latency should distinguish successful from failed requests, since fast failures skew averages misleadingly.

Error budget

The allowed unreliability derived as 1 minus the SLO. A 99.9% SLO over 30 days permits roughly 43 minutes of failure; exhausting the budget typically freezes feature releases.

Cardinality

The number of distinct label-value combinations across series. High-cardinality labels such as user ID or raw URL multiply series count, inflating memory, scrape size, and query cost.

USE versus RED

USE (Utilization, Saturation, Errors) targets resources like CPU and disks. RED (Rate, Errors, Duration) targets request-serving services. Choose based on whether you monitor infrastructure or endpoints.

Histogram versus summary

Histograms bucket observations server-side and aggregate across instances with histogram_quantile. Summaries compute quantiles client-side per instance and cannot be meaningfully aggregated, making histograms preferred for fleet-wide latency.

Pull-based scraping

Prometheus initiates HTTP GET requests to each target's metrics endpoint on a schedule. This makes target reachability directly observable through the up metric rather than relying on agents to push.

scrape_interval and scrape_timeout

scrape_interval sets how often a target is scraped, defaulting to one minute. scrape_timeout defaults to ten seconds and must be less than the interval, otherwise the configuration is rejected.

job and instance labels

Prometheus attaches job, taken from job_name, and instance, defaulting to the target's host:port, to every scraped series. These labels identify the source during queries and alerting.

relabel_configs versus metric_relabel_configs

relabel_configs rewrite target metadata before scraping and can drop targets; labels beginning with double underscore disappear afterwards. metric_relabel_configs act on scraped samples, commonly dropping noisy metrics.

Default retention

Prometheus keeps local TSDB data for 15 days by default, configurable with --storage.tsdb.retention.time or capped by size. Longer retention usually requires remote write to a separate store.

up metric

A synthetic gauge emitted per target: 1 when the most recent scrape succeeded, 0 when it failed. Alerting on up == 0 detects unreachable exporters and misconfigured service discovery.

Pushgateway purpose

A component that accepts pushed metrics for short-lived batch or cron jobs that cannot be scraped. It never expires pushed series automatically and should not be used for long-running services.

Remote write

A configuration block that streams samples from Prometheus to a remote endpoint such as Thanos, Mimir, or Cortex. It enables long-term retention and global querying because Prometheus itself is not clustered.

rate function

rate(counter[5m]) computes the per-second average increase of a counter over the range, requiring at least two samples. It extrapolates to range boundaries, so short or sparse ranges produce noisy results.

irate versus rate

irate uses only the last two samples in the range, giving an instant, spiky rate useful for fast-moving counters. rate averages over the whole range and is smoother, preferred for alerting and graphs.

increase function

increase(counter[5m]) returns the total counter growth over the range, approximately rate multiplied by range seconds. It is syntactic sugar over rate and inherits the same extrapolation behavior.

histogram_quantile

A function estimating a quantile from cumulative histogram buckets, requiring the le label. Aggregate with sum by (le) first; results are interpolated within buckets and are approximate, not exact.

by and without

Aggregation modifiers controlling which labels survive. by (job) keeps only the listed labels; without (instance) drops the listed labels and keeps the rest. They are mutually exclusive in one aggregation.

Vector matching on and ignoring

Binary operators between instant vectors match on all labels by default. on(...) restricts matching to listed labels; ignoring(...) excludes them. group_left or group_right permits many-to-one joins.

offset modifier

offset 1h shifts a query's evaluation time one hour into the past, allowing comparison of current values with earlier periods. It applies per selector, not to the whole expression.

@ modifier

The @ modifier pins a query to an absolute Unix timestamp, or to start() and end() of the query range. It makes expressions evaluate consistently regardless of when they run.

absent function

absent(metric) returns a single sample with value 1 when the selector matches no series, and nothing otherwise. It is used to alert on missing metrics rather than on bad values.

Counter metric type

A cumulative metric that only increases or resets to zero on restart. Counters should be queried with rate or increase; graphing raw counter values produces misleadingly rising lines.

Gauge metric type

A metric representing a value that can rise and fall arbitrarily, such as temperature or queue depth. Use delta, deriv, or raw instant queries rather than rate on gauges.

Summary metric type

A client-side metric exposing precomputed quantiles plus _sum and _count. Quantiles are calculated per instance and cannot be aggregated across instances, unlike histogram buckets.

node_exporter

The standard exporter for Linux host metrics such as CPU, memory, filesystem, and network, served on /metrics. Individual collectors can be toggled with --collector flags to reduce load.

blackbox_exporter

An exporter that probes endpoints over HTTP, HTTPS, TCP, ICMP, or DNS and exposes results like probe_success. Prometheus scrapes it with a target parameter in the relabeled URL.

Text exposition format

The plain-text format Prometheus scrapes: one sample per line as metric{label="value"} number, optionally followed by a millisecond timestamp. HELP and TYPE comment lines precede each metric family.

Naming conventions

Metric names use base units such as seconds and bytes, counters end in _total, and histograms expose _bucket, _sum, and _count. Labels carry dimensions and never unbounded identifiers.

Alerting rule for clause

The for field requires an alert expression to remain true continuously for a duration before firing. It moves alerts through inactive, pending, and firing states, suppressing transient spikes.

Alertmanager grouping

Alertmanager combines related alerts into a single notification using group_by labels, with group_wait, group_interval, and repeat_interval controlling timing. This reduces notification noise during incidents.

Inhibition versus silence

Inhibition automatically suppresses alerts when a related higher-severity alert is firing. Silences are manually created, time-bounded matchers that mute matching alerts regardless of other alerts.

Recording rules

Rules that periodically evaluate an expression and store the result as a new series, reducing query cost for dashboards. They follow the level:metric:operations naming convention such as job:http_requests:rate5m.

Annotation templating

Alert annotations support Go templates referencing $labels and $value, letting messages include the firing instance and threshold. Annotations are for humans; labels drive routing and grouping.