Observability Concepts
Observability Concepts tests whether you can choose the right signals, define reliability targets, and reason about the cost of measurement. Prometheus is a metrics system, so expect questions on where metrics fit beside logs and traces, and why label dimensions drive both usefulness and expense.
Signals and targets
- Metrics aggregate cheaply and alert well; logs preserve event detail; traces show request paths across services. Correlate them with shared labels such as service, environment, and trace ID.
- An SLI is a measured ratio or latency; an SLO is the target for that SLI over a window; an SLA is a contractual commitment. Error budget = 1 - SLO.
- Four golden signals: latency, traffic, errors, saturation. USE covers utilization, saturation, and errors for resources.
- Alert on symptoms users feel, such as error rate and latency, not every cause metric.
Worked example
An API SLO is 99.9% successful requests over 30 days. The error budget is 0.1% of 43,200 minutes = 43.2 minutes of full-outage equivalent. If the SLI is successful requests / total requests, compute it from a counter ratio over a suitable window, not from a monthly average of daily percentages. A burn-rate alert can fire faster when the budget is consumed quickly.
Traps
- Averaging per-instance percentiles or ratios without weighting by request count.
- Treating an SLA as an internal SLO, or setting an SLO without a measurement window.
- Adding user IDs or request IDs as labels, causing cardinality explosion.
- Calling every dashboard observability while lacking correlation IDs.