PCA: Prometheus Fundamentals

Prometheus Fundamentals

This domain covers the server architecture and configuration: pull-based scraping, service discovery, the TSDB, job and instance labels, retention, and the roles of Alertmanager and Pushgateway. Questions often test what Prometheus does by default versus what you must configure.

Core rules

• Prometheus scrapes HTTP targets on /metrics at scrape_interval; it does not wait for agents to push. A target is identified by job and instance labels.
• The data model is a metric name plus a label set; each sample has a timestamp and a float value. Series identity is the full label set.
• Scrape configs live under scrape_configs; static_configs is the simplest discovery, while file, DNS, Kubernetes, and cloud discovery are alternatives.
• evaluation_interval controls rule evaluation, not scraping. Alertmanager handles routing, grouping, inhibition, and silences; Prometheus evaluates rules and sends alerts.
• Pushgateway is for short-lived batch jobs that cannot be scraped; it is not a general remote-write proxy.

Worked example

A scrape config has job_name: api and static_configs: targets: ['10.0.0.5:9100']. Prometheus assigns job="api" and instance="10.0.0.5:9100" by default. If you add labels: {env: "prod"}, every series from that target also carries env="prod". Reasoning: target labels are attached after discovery and can be rewritten with relabel_configs, which run before scraping; metric_relabel_configs run after scraping and can drop samples.

Traps

Assuming Prometheus pushes metrics; using scrape_interval to control alert evaluation; forgetting that a down target still produces up=0; expecting deleted series to disappear instantly, since staleness markers take time; using Pushgateway for long-running services.