LF Observability common mistakes
LF Observability · Understand the traps and how to avoid them.
Treating the three pillars as interchangeable and answering that metrics can carry per-request identifiers or that traces are the cheapest way to track long-term trends.
Candidates learn the pillars as a list to memorize rather than as data types with distinct cost and cardinality profiles, so they default to whichever pillar they use most at work.
What to do instead
Build a decision table before the exam: metrics are pre-aggregated numeric time series with low-cardinality labels and cheap long retention; logs are high-detail discrete events, expensive at volume and usually sampled or tiered; traces are sampled request-scoped causal graphs suited to latency and dependency analysis. For each practice question, ask which pillar the described requirement (cost, cardinality, causality, retention) actually fits.
Confusing the three signal types when asked about SLIs, SLOs, and error budgets, for example computing an error budget from uptime of the monitoring agent rather than from the user-facing service indicator.
The vocabulary overlaps with infrastructure monitoring habits, and candidates who have only done host alerting map SLO concepts onto the wrong measurement point.
What to do instead
Practice writing an SLI as a ratio of good events to valid events measured as close to the user as possible, then derive the error budget as 1 minus the SLO target over the compliance window. Drill the distinction between SLI (measurement), SLO (target), SLA (contract with consequences), and error budget (permitted unreliability).
Assuming a histogram or summary metric answers arbitrary percentile questions across dimensions, then being surprised that percentiles cannot be averaged or re-aggregated after the fact.
Candidates confuse per-instance percentile computation with cross-instance aggregation and forget that summaries compute quantiles client-side and cannot be merged.
What to do instead
Memorize the trade-off: summaries give accurate quantiles per instance but are not aggregatable across instances; classic histograms are aggregatable only approximately and lose accuracy with coarse buckets; native or exponential histograms give configurable relative error and are the recommended choice in modern Prometheus and OpenTelemetry setups. When a question asks for fleet-wide p99, the correct answer is a histogram aggregated across series, not averaged summary quantiles.
Answering that adding a user ID, request ID, or full URL path as a metric label is good practice for debugging.
Labeling feels like adding context, and candidates carry over log-field habits without considering the series explosion each unique value creates in the time-series database.
What to do instead
State the rule explicitly: label values must come from a small bounded set. High-cardinality identifiers belong in logs or trace attributes, or in exemplars attached to histogram buckets. Recognize the symptoms of cardinality explosion (memory growth, slow queries, scrape timeouts) and the fixes (dropping or relabeling labels, aggregation at ingest, recording rules).
Treating log levels as decoration, answering that DEBUG belongs in production by default or that all ERROR logs should page a human.
Candidates conflate verbosity with observability and have not internalized that log volume drives cost and that many ERROR lines are expected and non-actionable.
What to do instead
Reason from the consumer: DEBUG for local development and short-lived incident triage with dynamic level changes; INFO for lifecycle events worth retaining; WARN for degraded but handled conditions; ERROR for failures needing attention but not necessarily paging. Alerts should key off SLO burn or user-visible symptoms, not raw log-level counts. Also recognize structured logging (key-value fields, stable field names) as the enabler for querying and sampling.
Mixing up trace structure terms, saying a span is the whole request, that trace IDs change per span, or that parent-child relationships are optional metadata.
Candidates who have only seen trace UIs have not internalized the data model: one trace ID shared by all spans, each span with its own span ID and a parent span ID forming the causal tree.
What to do instead
Draw the model once: trace equals the end-to-end operation identified by a trace ID; each unit of work is a span with span ID, parent span ID, start time, duration, status, and attributes; the root span has no parent. Then practice reading a described waterfall: identify the critical path, the slowest child, and where context propagation broke (a child appearing as a separate root).
Believing that adding the OpenTelemetry SDK automatically instruments libraries and exports data, and forgetting to register the tracer, meter, and logger providers or to configure an exporter and resource.
The API and SDK are separate packages, and a no-op tracer provider silently swallows spans with no error, so the code looks correct and produces nothing.
What to do instead
Memorize the initialization sequence: create a resource describing the service (service.name is mandatory and is what backends group by), build a tracer, meter, and logger provider with that resource, register a span processor with a batch exporter, and set the global providers before any instrumentation runs. Verify by checking that spans appear with the expected service name and that the batch processor flushes on shutdown.
Answering that the OpenTelemetry Collector's agent and gateway roles are interchangeable, or placing a tail-sampling processor on a DaemonSet agent.
The pipeline looks uniform in configuration files, and candidates miss that tail sampling requires seeing all spans of a trace in one place, which a per-node agent cannot guarantee.
What to do instead
Separate the roles: agents run close to workloads (sidecar or DaemonSet) for collection, batching, and light processing; gateways are centrally scaled for tail sampling, aggregation, and fan-out to multiple backends. Remember the processor ordering rule: memory_limiter first, batch last, and tail_sampling only where the whole trace is visible. Also know that receivers, processors, and exporters are wired per pipeline per signal type.
Assuming context propagation is automatic across every boundary, so spans from an upstream service and a downstream queue consumer appear as unrelated traces.
Propagation requires injecting and extracting the W3C traceparent header (or equivalent) at each hop, and messaging systems, background jobs, and non-instrumented proxies break the chain silently.
What to do instead
Trace a request across each boundary type in practice: HTTP client and server instrumentation handle headers; message producers must inject context into message attributes and consumers must extract it and start a span with a link or parent; batch jobs need explicit context creation. When debugging, look for orphan root spans and check whether the intermediary forwards the traceparent header.
Designing alerts and dashboards around every available metric, producing alert fatigue and dashboards nobody reads during an incident.
Tooling makes it easy to chart everything, and candidates equate coverage with quality instead of designing for the questions asked during an incident.
What to do instead
Tie alerts to SLO burn rates and user-visible symptoms, with runbooks and clear ownership; keep cause-based alerts for known imminent failures only. Structure dashboards by audience and question: a top-level service health view (traffic, errors, latency, saturation), drill-downs into dependencies, and resource views. Include the operational loop of triage, mitigation, and post-incident review, and mention that high-cardinality exploration belongs in traces and logs rather than on always-on dashboards.