PCA common mistakes
PCA · Understand the traps and how to avoid them.
Describing metrics, logs, and traces as interchangeable or saying a histogram is a log type, rather than distinguishing the three pillars and their cost/query tradeoffs.
Candidates memorise the vocabulary list without building a mental model of when each signal is appropriate, so under time pressure they conflate observability signals with telemetry formats.
What to do instead
For each pillar write one sentence on what question it answers (metrics: aggregate trends and alerting; logs: discrete event detail; traces: request path and latency breakdown), then practise classifying scenarios such as 'p99 latency regression by endpoint' or 'single failing request payload' into the correct signal.
Treating the four golden signals or RED/USE as universal, then choosing the wrong one for the resource type in the question (e.g. applying latency/traffic framing to a disk saturation problem).
Frameworks are learned as slogans rather than as decision rules tied to request-driven services versus resources, so the mapping step is skipped.
What to do instead
Build a two-column table: request-driven services use latency, traffic, errors, saturation; resources use utilisation, saturation, errors. For each practice scenario, name the workload type first, then select the framework, and justify why the alternative framework fits worse.
Assuming cardinality is only about the number of metric names, and missing that label value combinations (especially unbounded ones like user IDs or URLs) multiply series count.
Cardinality is easy to state abstractly but candidates rarely compute series counts, so they underestimate how one high-churn label explodes memory and scrape cost.
What to do instead
Practise arithmetic: series = metric names x distinct label value combinations. Given a metric with 3 labels of 10, 50, and 10,000 values, compute the series count and explain which label must be dropped or bounded. Also practise identifying unbounded labels in sample instrumentation snippets.
Writing a PromQL selector without understanding that metric name and label matchers are ANDed, or expecting a bare metric name to match several related metrics.
Candidates coming from SQL expect pattern matching or OR semantics, and the instant-vector selector syntax looks deceptively simple.
What to do instead
Memorise that a selector matches exactly one metric name plus all listed label matchers, and that alternatives require explicit regex matchers such as {name=~"http_requests_(total|created)"} or separate expressions combined with or. Write five selectors and predict their result sets before testing them.
Confusing counter, gauge, histogram, and summary semantics, for example applying rate() to a gauge or expecting a histogram to expose quantiles directly without histogram_quantile().
The four types are memorised as definitions rather than as behavioural contracts about monotonicity, reset handling, and what queries are valid.
What to do instead
For each type state: can it go down, what does rate() do to it, and what aggregation is valid. Then practise rewriting a broken query on each type, e.g. rate(temperature_celsius[5m]) corrected to a gauge-appropriate delta/average, and a summary quantile replaced with histogram_quantile over _bucket series.
Using rate() with a range window shorter than the scrape interval, or applying rate() to a raw counter and expecting per-second output without a range vector.
The relationship between scrape interval, range window, and the number of samples available is invisible in the syntax, so queries return empty or noisy results that candidates misread as data problems.
What to do instead
Always choose a range window of at least four scrape intervals (e.g. 1m window for a 15s scrape), and remember rate() requires a range vector. Practise explaining why rate(x[30s]) with a 60s scrape yields no samples, and check the scrape interval before writing any rate().
Ignoring counter resets and extrapolation when interpreting rate()/increase() output, then treating small negative or fractional values as bugs.
Candidates assume counters are always monotonic in the stored data, forgetting that process restarts and Prometheus's extrapolation logic both alter results.
What to do instead
Learn that rate() handles resets by assuming a reset when a sample decreases, and that increase() extrapolates to the window boundaries so results may be fractional. Practise reading a sample series with an embedded reset and hand-computing the expected rate before comparing with the query output.
Aggregating before or after the wrong operation, e.g. sum(rate(x[5m])) versus rate(sum(x)[5m]), and getting wrong per-instance or per-route totals.
The order of aggregation and rate changes both the meaning and validity of the result, and candidates copy patterns without reasoning about what each grouping preserves.
What to do instead
Adopt the rule: rate first on each series, then aggregate with sum by (...). Practise writing sum by (route) (rate(http_requests_total[5m])) and explain why applying sum before rate across a counter loses per-series reset information and can produce invalid range vectors.
Instrumenting with high-cardinality or unstable labels such as user_id, request_id, full URL path, or raw error messages, and creating metrics inside request handlers without control.
Instrumentation is written to be helpful for debugging rather than to respect the metric model, and label values look harmless in a single test request.
What to do instead
Apply a label checklist: bounded, low-cardinality, known at design time, and useful for aggregation. Replace IDs with status classes or route templates, and move per-request detail to logs or traces. Practise reviewing a code snippet and listing every label that would violate the checklist with a concrete replacement.
Forgetting that exporters and scrape targets are separate concerns, e.g. expecting the node_exporter to expose application metrics or assuming a missing target is a query problem.
The pull model and the target/endpoint distinction are abstract, so candidates debug at the PromQL layer when the failure is a scrape configuration or target health issue.
What to do instead
Trace the path explicitly: exporter exposes on /metrics, Prometheus scrapes the target per scrape_configs, then queries read stored samples. When a query is empty, check target health and up{} first. Practise diagnosing a scenario where up{job="app"} is 0 and deciding whether the fix is configuration, network, or query.
Writing alerting rules without a for duration or with an expression that fires on absent data, causing flapping alerts or silent gaps when a target disappears.
Candidates focus on the threshold expression and forget the lifecycle fields and the absent-data case, which only surface in production.
What to do instead
Always pair a threshold rule with a for duration sized to the scrape interval and evaluation interval, and add a separate rule using absent() or up{} == 0 for target loss. Practise writing a complete rule group with for, labels including severity, and annotations, then reason about how many evaluations must fail before firing.
Building dashboards and alerts on raw counters or instantaneous gauge samples instead of rates and sustained windows, producing spiky graphs and noisy pages.
Raw series are the easiest thing to plot and look plausible on a short time range, so the mismatch between instantaneous values and operational questions goes unnoticed.
What to do instead
Standardise on rate() or increase() with a window for counters and on aggregation with averaging or max_over_time for gauges. For each panel, state the operational question it answers and confirm the query's window matches the alert's for duration so the dashboard and the page agree.