LF Observability cheat sheet

Three pillars: metrics (aggregated numeric time series), logs (discrete timestamped event records), traces (causal span trees for a single request).

Observability is the ability to answer novel questions about internal state from external outputs without shipping new code; monitoring answers only pre-defined questions.

SLI is a measured ratio (e.g., successful requests / total requests); SLO is a target range for that SLI over a window; SLA is a contractual consequence-bearing commitment and is usually looser than the SLO.

Error budget = 1 − SLO. A 99.9% monthly SLO allows about 43.2 minutes of unavailability per 30-day month; 99.95% allows about 21.6 minutes.

Burn rate = observed error rate / error budget rate. Burn rate 1 exhausts the budget exactly at the end of the window; multi-window alerts pair a long window (e.g., 1h) with a short window (e.g., 5m) to reduce false positives.

Cardinality is the number of distinct label combinations; high-cardinality dimensions (user ID, request ID, full URL) multiply series count and are the main cost and performance driver in metrics backends.

Sampling trades completeness for cost: head sampling decides at span start (cheap, no trace knowledge), tail sampling decides after the trace completes (can keep all errors and slow traces).

Correlation requires shared identifiers: propagate trace_id and span_id into logs, and exemplars from metric histograms to trace IDs, so a spike can be pivoted to a specific request.

Percentiles are not averageable: you cannot average per-instance p99 values to get a fleet p99. Aggregate raw histograms or use native histogram/sketch types instead.

Prometheus metric types: counter (monotonic, use rate()/increase()), gauge (up/down, use raw value or delta), histogram (bucket counts plus _sum and _count, gives server-side quantiles via histogram_quantile), summary (client-side quantiles, not aggregatable across instances).

Counter reset handling: rate() and increase() in PromQL detect resets and extrapolate; applying rate() to a gauge produces meaningless values.

PromQL rate(x[5m]) returns per-second average over the window; increase(x[5m]) returns the total increase over roughly the window length, and both extrapolate to window boundaries.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes a fleet p95; the le label must be preserved in the aggregation.

Trace model: a trace is a DAG of spans; each span has trace_id, span_id, parent_span_id, start time, duration, status, and attributes. The root span has no parent.

Context propagation uses W3C traceparent (version-traceid-spanid-flags) and tracestate headers; a broken propagation link creates a new root span and splits one logical trace into several.

Span status codes in OTel: UNSET (default, not an error), OK (explicitly successful), ERROR. Sampling decisions and errors are separate concerns.

Structured logs in JSON with stable field names (timestamp, level, service, trace_id, span_id) are queryable; unstructured text requires regex parsing at query time and is fragile.

Log levels are severity filters, not storage tiers: sampling and retention policies (e.g., keep errors 30 days, debug 3 days) control cost more than level choice alone.

OTel signals: traces, metrics, logs, plus baggage and context. The API is used by instrumented libraries; the SDK implements it and is configured by the application or the Collector.

Resource describes the entity producing telemetry (service.name, service.version, host.name, deployment.environment) and is set once per SDK/process; attributes describe individual spans, metrics, or log records.

Instrumentation scope: automatic instrumentation (agents/instrumentation libraries) versus manual instrumentation via the API (tracer.Start, span attributes, status).

OTLP is the wire protocol (gRPC on port 4317, HTTP/protobuf on port 4318 by default). Receivers, processors, exporters, and connectors form Collector pipelines defined in YAML under service.pipelines.

Collector deployment patterns: agent (DaemonSet, per-node) for local collection and enrichment, gateway (deployment, centralized) for batching, sampling, and fan-out to multiple backends.

Key processors: batch (groups and compresses before export), memory_limiter (drops data before OOM), attributes/resource (add, update, delete, hash), tail_sampling (policy-based keep/drop after trace completion), filter (drop by condition).

Tail sampling requires all spans of a trace to reach the same Collector instance; route by trace_id using the loadbalancing exporter with routing_key: traceID.

Semantic conventions standardize attribute names (http.request.method, http.response.status_code, db.system, server.address) so dashboards and alerts work across services and languages.

Environment variables configure autoconfiguration: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES, OTEL_SDK_DISABLED.

Retention tiers: recent data on fast storage for interactive queries, older data downsampled or on object storage for compliance and trend analysis; downsampling reduces resolution, not correctness of counters.

Alerting rule design: alert on symptoms (SLO burn rate, error ratio, latency) rather than causes (CPU high); include a runbook link, severity, and for/how long to avoid flapping.

Multi-window burn-rate alert example: page when burn rate > 14.4 over 1h and > 14.4 over 5m (consumes 2% of a 30-day budget in an hour); ticket when burn rate > 6 over 6h.

Golden signals: latency (distinguish successful vs failed request latency), traffic, errors, saturation. USE (utilization, saturation, errors) fits resources; RED (rate, errors, duration) fits services.

Cardinality control: drop or hash unbounded labels at the Collector, aggregate with recording rules, and prefer bounded dimensions (status code class, route template) over raw paths and IDs.

Query cost control: recording rules precompute expensive PromQL; avoid high-cardinality group-by in dashboards; use step sizes matched to retention resolution to avoid over-fetching.

Exemplars attach a trace_id to a histogram bucket so a latency spike on a dashboard links directly to a representative trace for root-cause analysis.

Incident workflow: detect (alert) → triage (dashboards, golden signals) → localize (traces, service map) → diagnose (logs with trace_id) → mitigate → write a blameless postmortem with action items.

Data quality checks: monitor telemetry pipeline health (Collector refused/dropped spans, exporter queue length, scrape target up==0) because missing telemetry looks identical to a healthy system.

Three pillars: metrics (aggregated numeric time series), logs (discrete timestamped event records), traces (causal span trees for a single request).

Observability is the ability to answer novel questions about internal state from external outputs without shipping new code; monitoring answers only pre-defined questions.

SLI is a measured ratio (e.g., successful requests / total requests); SLO is a target range for that SLI over a window; SLA is a contractual consequence-bearing commitment and is usually looser than the SLO.

Error budget = 1 − SLO. A 99.9% monthly SLO allows about 43.2 minutes of unavailability per 30-day month; 99.95% allows about 21.6 minutes.

Burn rate = observed error rate / error budget rate. Burn rate 1 exhausts the budget exactly at the end of the window; multi-window alerts pair a long window (e.g., 1h) with a short window (e.g., 5m) to reduce false positives.

Cardinality is the number of distinct label combinations; high-cardinality dimensions (user ID, request ID, full URL) multiply series count and are the main cost and performance driver in metrics backends.

Sampling trades completeness for cost: head sampling decides at span start (cheap, no trace knowledge), tail sampling decides after the trace completes (can keep all errors and slow traces).

Correlation requires shared identifiers: propagate trace_id and span_id into logs, and exemplars from metric histograms to trace IDs, so a spike can be pivoted to a specific request.

Percentiles are not averageable: you cannot average per-instance p99 values to get a fleet p99. Aggregate raw histograms or use native histogram/sketch types instead.

Prometheus metric types: counter (monotonic, use rate()/increase()), gauge (up/down, use raw value or delta), histogram (bucket counts plus _sum and _count, gives server-side quantiles via histogram_quantile), summary (client-side quantiles, not aggregatable across instances).

Counter reset handling: rate() and increase() in PromQL detect resets and extrapolate; applying rate() to a gauge produces meaningless values.

PromQL rate(x[5m]) returns per-second average over the window; increase(x[5m]) returns the total increase over roughly the window length, and both extrapolate to window boundaries.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes a fleet p95; the le label must be preserved in the aggregation.

Trace model: a trace is a DAG of spans; each span has trace_id, span_id, parent_span_id, start time, duration, status, and attributes. The root span has no parent.

Context propagation uses W3C traceparent (version-traceid-spanid-flags) and tracestate headers; a broken propagation link creates a new root span and splits one logical trace into several.

Span status codes in OTel: UNSET (default, not an error), OK (explicitly successful), ERROR. Sampling decisions and errors are separate concerns.

Structured logs in JSON with stable field names (timestamp, level, service, trace_id, span_id) are queryable; unstructured text requires regex parsing at query time and is fragile.

Log levels are severity filters, not storage tiers: sampling and retention policies (e.g., keep errors 30 days, debug 3 days) control cost more than level choice alone.

OTel signals: traces, metrics, logs, plus baggage and context. The API is used by instrumented libraries; the SDK implements it and is configured by the application or the Collector.

Resource describes the entity producing telemetry (service.name, service.version, host.name, deployment.environment) and is set once per SDK/process; attributes describe individual spans, metrics, or log records.

Instrumentation scope: automatic instrumentation (agents/instrumentation libraries) versus manual instrumentation via the API (tracer.Start, span attributes, status).

OTLP is the wire protocol (gRPC on port 4317, HTTP/protobuf on port 4318 by default). Receivers, processors, exporters, and connectors form Collector pipelines defined in YAML under service.pipelines.

Collector deployment patterns: agent (DaemonSet, per-node) for local collection and enrichment, gateway (deployment, centralized) for batching, sampling, and fan-out to multiple backends.

Key processors: batch (groups and compresses before export), memory_limiter (drops data before OOM), attributes/resource (add, update, delete, hash), tail_sampling (policy-based keep/drop after trace completion), filter (drop by condition).

Tail sampling requires all spans of a trace to reach the same Collector instance; route by trace_id using the loadbalancing exporter with routing_key: traceID.

Semantic conventions standardize attribute names (http.request.method, http.response.status_code, db.system, server.address) so dashboards and alerts work across services and languages.

Environment variables configure autoconfiguration: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES, OTEL_SDK_DISABLED.

Retention tiers: recent data on fast storage for interactive queries, older data downsampled or on object storage for compliance and trend analysis; downsampling reduces resolution, not correctness of counters.

Alerting rule design: alert on symptoms (SLO burn rate, error ratio, latency) rather than causes (CPU high); include a runbook link, severity, and for/how long to avoid flapping.

Multi-window burn-rate alert example: page when burn rate > 14.4 over 1h and > 14.4 over 5m (consumes 2% of a 30-day budget in an hour); ticket when burn rate > 6 over 6h.

Golden signals: latency (distinguish successful vs failed request latency), traffic, errors, saturation. USE (utilization, saturation, errors) fits resources; RED (rate, errors, duration) fits services.

Cardinality control: drop or hash unbounded labels at the Collector, aggregate with recording rules, and prefer bounded dimensions (status code class, route template) over raw paths and IDs.

Query cost control: recording rules precompute expensive PromQL; avoid high-cardinality group-by in dashboards; use step sizes matched to retention resolution to avoid over-fetching.

Exemplars attach a trace_id to a histogram bucket so a latency spike on a dashboard links directly to a representative trace for root-cause analysis.

Incident workflow: detect (alert) → triage (dashboards, golden signals) → localize (traces, service map) → diagnose (logs with trace_id) → mitigate → write a blameless postmortem with action items.

Data quality checks: monitor telemetry pipeline health (Collector refused/dropped spans, exporter queue length, scrape target up==0) because missing telemetry looks identical to a healthy system.

Three pillars: metrics (aggregated numeric time series), logs (discrete timestamped event records), traces (causal span trees for a single request).

Observability is the ability to answer novel questions about internal state from external outputs without shipping new code; monitoring answers only pre-defined questions.

SLI is a measured ratio (e.g., successful requests / total requests); SLO is a target range for that SLI over a window; SLA is a contractual consequence-bearing commitment and is usually looser than the SLO.

Error budget = 1 − SLO. A 99.9% monthly SLO allows about 43.2 minutes of unavailability per 30-day month; 99.95% allows about 21.6 minutes.

Burn rate = observed error rate / error budget rate. Burn rate 1 exhausts the budget exactly at the end of the window; multi-window alerts pair a long window (e.g., 1h) with a short window (e.g., 5m) to reduce false positives.

Cardinality is the number of distinct label combinations; high-cardinality dimensions (user ID, request ID, full URL) multiply series count and are the main cost and performance driver in metrics backends.

Sampling trades completeness for cost: head sampling decides at span start (cheap, no trace knowledge), tail sampling decides after the trace completes (can keep all errors and slow traces).

Correlation requires shared identifiers: propagate trace_id and span_id into logs, and exemplars from metric histograms to trace IDs, so a spike can be pivoted to a specific request.

Percentiles are not averageable: you cannot average per-instance p99 values to get a fleet p99. Aggregate raw histograms or use native histogram/sketch types instead.

Prometheus metric types: counter (monotonic, use rate()/increase()), gauge (up/down, use raw value or delta), histogram (bucket counts plus _sum and _count, gives server-side quantiles via histogram_quantile), summary (client-side quantiles, not aggregatable across instances).

Counter reset handling: rate() and increase() in PromQL detect resets and extrapolate; applying rate() to a gauge produces meaningless values.

PromQL rate(x[5m]) returns per-second average over the window; increase(x[5m]) returns the total increase over roughly the window length, and both extrapolate to window boundaries.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) computes a fleet p95; the le label must be preserved in the aggregation.

Trace model: a trace is a DAG of spans; each span has trace_id, span_id, parent_span_id, start time, duration, status, and attributes. The root span has no parent.

Context propagation uses W3C traceparent (version-traceid-spanid-flags) and tracestate headers; a broken propagation link creates a new root span and splits one logical trace into several.

Span status codes in OTel: UNSET (default, not an error), OK (explicitly successful), ERROR. Sampling decisions and errors are separate concerns.

Structured logs in JSON with stable field names (timestamp, level, service, trace_id, span_id) are queryable; unstructured text requires regex parsing at query time and is fragile.

Log levels are severity filters, not storage tiers: sampling and retention policies (e.g., keep errors 30 days, debug 3 days) control cost more than level choice alone.

OTel signals: traces, metrics, logs, plus baggage and context. The API is used by instrumented libraries; the SDK implements it and is configured by the application or the Collector.

Resource describes the entity producing telemetry (service.name, service.version, host.name, deployment.environment) and is set once per SDK/process; attributes describe individual spans, metrics, or log records.

Instrumentation scope: automatic instrumentation (agents/instrumentation libraries) versus manual instrumentation via the API (tracer.Start, span attributes, status).

OTLP is the wire protocol (gRPC on port 4317, HTTP/protobuf on port 4318 by default). Receivers, processors, exporters, and connectors form Collector pipelines defined in YAML under service.pipelines.

Collector deployment patterns: agent (DaemonSet, per-node) for local collection and enrichment, gateway (deployment, centralized) for batching, sampling, and fan-out to multiple backends.

Key processors: batch (groups and compresses before export), memory_limiter (drops data before OOM), attributes/resource (add, update, delete, hash), tail_sampling (policy-based keep/drop after trace completion), filter (drop by condition).

Tail sampling requires all spans of a trace to reach the same Collector instance; route by trace_id using the loadbalancing exporter with routing_key: traceID.

Semantic conventions standardize attribute names (http.request.method, http.response.status_code, db.system, server.address) so dashboards and alerts work across services and languages.

Environment variables configure autoconfiguration: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES, OTEL_SDK_DISABLED.

Retention tiers: recent data on fast storage for interactive queries, older data downsampled or on object storage for compliance and trend analysis; downsampling reduces resolution, not correctness of counters.

Alerting rule design: alert on symptoms (SLO burn rate, error ratio, latency) rather than causes (CPU high); include a runbook link, severity, and for/how long to avoid flapping.

Multi-window burn-rate alert example: page when burn rate > 14.4 over 1h and > 14.4 over 5m (consumes 2% of a 30-day budget in an hour); ticket when burn rate > 6 over 6h.

Golden signals: latency (distinguish successful vs failed request latency), traffic, errors, saturation. USE (utilization, saturation, errors) fits resources; RED (rate, errors, duration) fits services.

Cardinality control: drop or hash unbounded labels at the Collector, aggregate with recording rules, and prefer bounded dimensions (status code class, route template) over raw paths and IDs.

Query cost control: recording rules precompute expensive PromQL; avoid high-cardinality group-by in dashboards; use step sizes matched to retention resolution to avoid over-fetching.

Exemplars attach a trace_id to a histogram bucket so a latency spike on a dashboard links directly to a representative trace for root-cause analysis.

Incident workflow: detect (alert) → triage (dashboards, golden signals) → localize (traces, service map) → diagnose (logs with trace_id) → mitigate → write a blameless postmortem with action items.

Data quality checks: monitor telemetry pipeline health (Collector refused/dropped spans, exporter queue length, scrape target up==0) because missing telemetry looks identical to a healthy system.