LF Observability flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Observability vs monitoring

Monitoring answers predefined questions such as whether a known metric crossed a threshold. Observability is the property that lets you answer unanticipated questions from telemetry without deploying new instrumentation code.

SLI, SLO, SLA

An SLI is a measured service ratio like successful requests divided by total requests. An SLO is a target range for that SLI over a window. An SLA is a contractual commitment with consequences, typically looser.

Error budget

Error budget equals one minus the SLO target. A 99.9 percent SLO over 30 days permits roughly 43 minutes of failure; the budget quantifies how much unreliability is acceptable for shipping velocity.

Burn rate

Burn rate is the observed error rate divided by the error-budget rate. A burn rate of one exhausts the budget exactly at window end; higher values exhaust it faster and justify paging.

Cardinality

Cardinality is the count of distinct label or attribute combinations for a metric. Unbounded values such as user IDs or raw URLs multiply series and drive storage and query cost.

Head vs tail sampling

Head sampling decides whether to record a trace at its start, is cheap, and cannot see the outcome. Tail sampling decides after the trace completes, so it can retain all errors and slow traces.

Exemplar

An exemplar is a sample value attached to a metric histogram bucket that carries a trace ID. It lets a dashboard latency spike link directly to a representative trace for diagnosis.

Percentile aggregation pitfall

Percentiles cannot be averaged. Averaging per-instance p99 values does not yield the fleet p99; you must aggregate raw histogram buckets or use a mergeable sketch type instead.

Prometheus counter

A counter is a monotonically increasing cumulative value, such as total requests. Query it with rate() or increase(), which detect counter resets and extrapolate to window boundaries.

Prometheus gauge

A gauge is a value that can rise or fall, such as memory usage or queue depth. Query the raw value or a delta; applying rate() to a gauge produces meaningless results.

Histogram vs summary

A histogram exposes cumulative bucket counts plus sum and count, allowing server-side quantiles and cross-instance aggregation. A summary computes quantiles client-side per instance and cannot be aggregated across instances.

histogram_quantile usage

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes a fleet p95. The le bucket label must be preserved in the aggregation or the result is wrong.

rate versus increase

rate(x[5m]) returns the average per-second increase over the window. increase(x[5m]) returns the total increase over roughly the same window. Both handle counter resets and extrapolate.

Trace and span model

A trace is a directed acyclic graph of spans for one request. Each span carries trace_id, span_id, parent_span_id, timing, status, and attributes; the root span has no parent.

W3C traceparent header

The traceparent header encodes version, trace ID, parent span ID, and trace flags. It carries context across service boundaries; a missing or broken header starts a new trace and splits causality.

OTel span status

Span status is UNSET by default and does not mean success or failure. OK marks explicit success and ERROR marks failure. Sampling decisions are recorded separately in trace flags.

Structured logging

Structured logs emit JSON with stable field names such as timestamp, level, service, trace_id, and span_id. They are directly queryable, unlike free text requiring fragile regex parsing.

Log retention tiering

Retention policies keep high-severity logs longer and drop or sample debug logs sooner. Log level is a severity filter, not a storage policy; retention and sampling control cost.

OTel API vs SDK

The OpenTelemetry API is the instrumentation surface libraries call, such as starting spans. The SDK implements it and is configured by the application or Collector for exporters, sampling, and resources.

OTel resource attributes

A resource describes the entity producing telemetry: service.name, service.version, host.name, and deployment.environment. It is set once per process and attached to all emitted signals.

OTLP default ports

OTLP uses gRPC on port 4317 and HTTP with protobuf on port 4318 by default. These are the standard endpoints applications and Collectors target when exporting telemetry.

Collector pipeline structure

A Collector pipeline combines receivers that accept data, processors that transform or batch it, and exporters that send it onward. Pipelines are declared per signal under service.pipelines in YAML.

Agent versus gateway Collector

An agent Collector runs per node, often as a DaemonSet, for local collection and enrichment. A gateway Collector is centralized and handles batching, tail sampling, and fan-out to backends.

Batch processor

The batch processor groups spans, metrics, or logs and compresses them before export. It reduces network requests and backend load; it is usually placed after memory_limiter in a pipeline.

memory_limiter processor

The memory_limiter processor checks memory usage and refuses or drops data before the Collector runs out of memory. It should be first in the pipeline to protect the process.

Tail sampling routing

Tail sampling requires every span of a trace to reach the same Collector. Route by trace ID using the loadbalancing exporter with routing_key traceID so decisions see complete traces.

Semantic conventions

Semantic conventions standardize attribute names such as http.request.method, http.response.status_code, and db.system. Shared names let dashboards, alerts, and queries work across services and languages.

OTEL_SERVICE_NAME variable

OTEL_SERVICE_NAME sets the service.name resource attribute used by SDK autoconfiguration. It identifies the service in traces, metrics, and logs and drives service maps and filtering.

Symptom-based alerting

Symptom-based alerts fire on user-visible conditions such as error ratio or latency, not causes like high CPU. Cause-based alerts generate noise because many causes never affect users.

Golden signals

The four golden signals are latency, traffic, errors, and saturation. Latency should separate successful from failed requests, since fast failures can hide behind favorable averages.

USE and RED methods

USE covers utilization, saturation, and errors for resources like CPU and disks. RED covers rate, errors, and duration for request-serving services. Choose by whether you monitor a resource or a service.

Recording rules

Recording rules precompute expensive or frequently used PromQL expressions and store results as new series. Dashboards and alerts then query cheap precomputed values instead of scanning raw data.