Prometheus Certified Associate cheat sheet

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.

The three pillars are metrics (numeric time series), logs (discrete events), and traces (request paths); metrics are cheapest to store and aggregate at scale.

Four Golden Signals (Google SRE): latency, traffic, errors, saturation. Latency should separate successful from failed request durations.

USE method (Gregg): for every resource check Utilization, Saturation, Errors - best for hosts and infrastructure.

RED method: Rate, Errors, Duration - best for request-serving services and APIs.

SLI is a measured ratio (e.g., good requests / valid requests); SLO is a target for that SLI (e.g., 99.9% over 30 days); SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. At 99.9% over 30 days, budget is ~43.2 minutes of allowed failure.

Pull-based scraping makes target health observable: a failed scrape yields up == 0 rather than silent gaps.

Cardinality is the product of distinct label-value combinations; high-cardinality labels (user IDs, request IDs, raw URLs) cause memory and query cost blowups.

A histogram approximates a distribution using cumulative buckets plus _sum and _count, enabling server-side quantile estimation via histogram_quantile.

Architecture: Prometheus server (retrieval, TSDB, HTTP API), Alertmanager (separate binary), exporters, Pushgateway, and service discovery. Prometheus does not store logs or traces.

Scrape config essentials: job_name, scrape_interval (default 1m), scrape_timeout (default 10s, must be < interval), metrics_path (default /metrics), scheme (default http), static_configs targets.

Every scraped series gets job and instance labels; instance defaults to the target's __address__ (host:port).

Relabeling runs in order: relabel_configs on target metadata before scraping (labels prefixed __ are dropped afterwards), metric_relabel_configs on samples after scraping.

Storage: local TSDB blocks of ~2h, WAL for crash recovery, retention by time (--storage.tsdb.retention.time, default 15d) or size.

Prometheus is not clustered; for long-term retention and global querying, remote write to a remote store (e.g., Thanos, Mimir, Cortex) is used.

up is a synthetic gauge: 1 if the last scrape succeeded, 0 otherwise; scrape_duration_seconds and scrape_samples_scraped expose scrape health.

Pushgateway is for short-lived batch jobs only; pushed metrics are never expired automatically and it loses the instance label semantics of scraping.

Service discovery (Kubernetes, Consul, EC2, DNS, file_sd) generates targets; label sets like __meta_kubernetes_pod_label_* are available only during relabeling.

Selector syntax: metric{label="value",label2=~"regex"} with matchers =, !=, =~, !~. At least one non-empty matcher is required; regexes are fully anchored.

Range vector metric[5m] is only valid inside functions (e.g., rate, avg_over_time); instant vector is the default result type.

rate() requires a range vector of a counter and needs at least two samples; it extrapolates to a per-second value. Use irate() for the last two samples (spiky, fast-moving counters).

increase(v[5m]) ≈ rate(v[5m]) * 300; delta() is for gauges, idelta() for the last two gauge samples.

Aggregation: sum, avg, min, max, count, stddev, topk, bottomk, quantile with by (labels) or without (labels). topk keeps original series labels.

Binary operators between instant vectors match on all labels by default; use on(...)/ignoring(...) plus group_left/group_right for many-to-one joins.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) estimates the 95th percentile; le must be preserved in aggregation.

offset 1h shifts the query evaluation time; @ 1609746000 (or @ start()/@ end()) pins evaluation to an absolute timestamp.

Absent data handling: absent(metric) returns 1 when the series is missing; clamp_min, vector(0), and or are common for filling gaps.

Client library metric types: Counter (monotonic, only increases), Gauge (goes up and down), Histogram (buckets + sum + count), Summary (client-side quantiles + sum + count, not aggregatable across instances).

Naming conventions: base units (seconds, bytes), suffix _total for counters, _bucket/_sum/_count for histograms; labels are for dimensions, never for high-cardinality IDs.

Histogram bucket boundaries are cumulative and must be chosen for expected latency ranges; default buckets in Go are 0.005s to 10s.

Exporters translate third-party metrics into Prometheus format over HTTP /metrics: node_exporter (host), blackbox_exporter (probe), mysqld_exporter, cAdvisor (containers).

The text exposition format is one sample per line: metric_name{label="value"} 1234 [timestamp_ms]; # HELP and # TYPE precede the metric family.

Pushgateway accepts pushes via POST /metrics/job/<job>; use it only for batch/cron jobs that cannot be scraped.

node_exporter collectors can be enabled/disabled with --collector.<name> and --no-collector.<name>; textfile collector ingests .prom files from a directory.

Instrumentation should measure at the request boundary (middleware) and use labels such as method, status, handler - not path with unbounded values.

Alerting rules live in rule files referenced by rule_files in prometheus.yml; each rule has alert, expr, optional for, labels, and annotations.

for: 5m requires the expression to be true continuously for 5 minutes before firing; state transitions are inactive → pending → firing.

annotations carry human-readable text using templating such as {{ $labels.instance }} and {{ $value }}; labels add routing metadata like severity.

Prometheus evaluates rules at evaluation_interval (default 1m) and pushes firing alerts to Alertmanager via alerting.alertmanagers.

Alertmanager handles deduplication, grouping (group_by), inhibition, and silencing, then routes to receivers (email, Slack, PagerDuty) via a routing tree.

Inhibition suppresses lower-severity alerts when a matching higher-severity alert fires; silences are time-bounded and matched by label matchers.

Alert on symptoms (error rate, latency SLO burn) rather than causes where possible; include a runbook URL and dashboard link in annotations.

Grafana queries Prometheus through a datasource; use variables ($instance), rate() for counters, and set dashboard time ranges consistent with retention.

Recording rules precompute expensive expressions into new series (e.g., job:http_requests:rate5m) and are named with the level:metric:operations convention.