Full-Stack Observability, Distributed Tracing, and Observability-Driven Development
Observability is the ability to ask new questions of a system without shipping new code. It is commonly decomposed into metrics, logs, and traces, but the modern framing emphasizes high-cardinality, high-dimensionality events (wide structured events) joined by shared context. Distributed tracing is the backbone of that join: a trace is a tree of spans sharing a trace ID, and context propagation (typically W3C Trace Context headers) must survive HTTP, gRPC, message queues, and async workers - dropped context in an async hop is one of the most common real-world gaps and a favourite exam trap. Sampling strategies matter: head-based sampling is cheap but can discard the rare errors you need; tail-based sampling keeps interesting traces (errors, high latency) at the cost of buffering.
Candidates should know the standard measurement frameworks: golden signals (latency, traffic, errors, saturation), RED (rate, errors, duration) for request-driven services, and USE (utilization, saturation, errors) for resources. OpenTelemetry is the vendor-neutral standard for instrumentation; structured logging with consistent fields beats free-text logs; exemplars link metrics to example traces.
Observability-driven development means instrumentation is written with the code, not bolted on after an incident: developers assert on telemetry in tests, use traces to review call paths, define SLIs at design time, and treat dashboards and alerts as code reviewed like any other artifact.
Traps: putting unbounded values such as user IDs or request IDs in metric labels (cardinality explosion); treating dashboards as proof of observability; relying on head sampling when investigating rare failures; measuring only server-side latency and ignoring client experience; and confusing distributed tracing with trace-log correlation.
Rule of thumb: every request should be answerable as "which trace, which spans, which log lines, which metric exemplar" - if a single failing user request cannot be reconstructed from telemetry alone, coverage is incomplete.