PCA: Instrumentation and Exporters study guide

Instrumentation and Exporters

This domain tests how metrics get produced: client libraries, metric types, naming, labels, exporters, and Pushgateway. Questions often ask which metric type fits a measurement or which exporter exposes a system.

Core rules

Counters only increase or reset; gauges go up and down; histograms bucket observations and expose _bucket, _sum, and _count; summaries expose quantiles plus _sum and _count.
Name counters with _total, use base units such as seconds and bytes, and keep labels low-cardinality. Put the measured thing in the metric name, not in a label value.
Client libraries instrument your code; exporters translate third-party systems. node_exporter exposes host metrics, blackbox_exporter probes endpoints, and mysqld_exporter exposes database metrics.
Pushgateway accepts pushes for batch jobs; Prometheus still scrapes the gateway. Do not use it for long-running services.

Worked example

A histogram http_request_duration_seconds with buckets 0.1, 0.5, 1, and 5 exposes http_request_duration_seconds_bucket{le="0.5"}. To estimate the 95th percentile across instances: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))). Reasoning: rate() turns each bucket into a per-second count, sum by (le) combines instances, and histogram_quantile interpolates within the bucket containing the 95th percentile. Dropping le breaks the function.

Traps

Using a gauge for total requests; creating a new counter per status code instead of a code label; adding user_id or URL path labels; aggregating summary quantiles; assuming an exporter instruments business logic; pushing long-running service metrics to Pushgateway and losing instance identity.