Observability Foundation cheat sheet

Three Pillars: metrics (aggregated numeric time series), logs (discrete timestamped event records), traces (causal span trees following a request across services).

Monitoring answers known-unknowns via predefined dashboards and alerts; observability answers unknown-unknowns by allowing arbitrary questions of high-cardinality, high-dimensionality telemetry without shipping new code.

Black-box monitoring observes symptoms from outside (synthetic probes, endpoint checks); white-box monitoring exposes internal state (queues, goroutine counts, cache hit ratios).

SLI is a measured ratio of good events to valid events; SLO is a target ratio over a window; SLA is a contractual consequence. Error budget = 1 − SLO.

Alert on symptoms that affect users (SLO burn rate), not on every cause; multi-window burn-rate alerts (e.g., 1h at 14.4x, 6h at 6x) reduce both false positives and detection lag.

Cardinality rule: label dimensions multiply series count; a metric with 5 labels of 100 values each yields 100^5 potential series, so unbounded labels (user ID, request ID) belong in traces/logs, not metrics.

Observability is a property of the system, not a tool: it requires instrumentation decisions made at design time and a culture where engineers can ask new questions post-deployment.

Toil reduction: each manually triaged alert that produces no action should be deleted or automated; track alert precision (actionable alerts / total alerts) as a first-class metric.

Telemetry cost scales with volume, cardinality, and retention; sampling, aggregation, and tiered retention are design levers, not afterthoughts.

OpenTelemetry (OTel) is a CNCF project providing vendor-neutral APIs, SDKs, and a wire protocol (OTLP) for traces, metrics, logs, and baggage.

Signals: traces, metrics, logs, baggage, and profiles (profiles are a newer, still-stabilizing signal). OTLP runs over gRPC (port 4317) or HTTP/protobuf (port 4318).

API vs SDK: the API is a no-op-safe interface used by instrumentation libraries; the SDK implements processing (samplers, processors, exporters) and is configured in application setup.

Resource describes the entity producing telemetry (service.name, service.version, host.name, k8s.pod.name, deployment.environment); service.name is required and defaults to 'unknown_service' if unset.

Context propagation uses W3C Trace Context (traceparent, tracestate headers) and W3C Baggage; propagators must be configured on both client and server or traces break at the boundary.

SpanKind values: INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER; choosing correctly is required for service-graph inference and latency attribution.

The Collector has receivers, processors (batch, memory_limiter, tail_sampling, attributes), exporters, and connectors; a standard pipeline order is memory_limiter, then batch, placed last before export.

Semantic conventions standardize attribute names (http.request.method, http.response.status_code, db.system, messaging.system); use them rather than custom names so backends auto-understand data.

OTel auto-instrumentation exists for common runtimes (Java agent, Python, Node, .NET, Go via eBPF or wrappers); manual instrumentation adds domain-specific spans and attributes.

A trace is a directed acyclic graph of spans sharing one trace_id; each span has a span_id and an optional parent_span_id; the root span has no parent.

A trace context travels in the traceparent header: version-trace_id-parent_span_id-flags; the sampled bit in flags tells downstream services whether to record.

Head-based sampling decides at the root before the trace is complete (cheap, but can miss rare errors); tail-based sampling decides after all spans arrive at the Collector (accurate, requires buffering by trace_id).

Critical path: the span path whose summed exclusive duration equals end-to-end latency; optimizing spans off the critical path does not improve response time.

Clock skew across hosts makes span start/end times unreliable; use parent-child relationships and span durations rather than absolute timestamps for causal analysis.

Missing spans usually indicate broken context propagation (async boundaries, message queues, thread pools, third-party calls) rather than a backend fault.

Span links (not parent-child) model causal relationships across asynchronous or batched work, e.g., one consumer span linked to many producer spans.

Service graphs and span metrics are derived from parent-child span pairs; they enable RED (rate, errors, duration) dashboards without per-service instrumentation.

Sampling consistency: propagate the sampling decision via trace flags so all services in a trace agree; otherwise partial traces appear.

Span fields: trace_id, span_id, parent_span_id, name, start/end timestamps, attributes (key-value), events (timestamped annotations), status, and links.

Span status is Unset, Ok, or Error; set Error only for failed operations, and record the exception via a span event with type, message, and stacktrace.

Attributes are indexed and queryable and should be low-cardinality; events are timestamped logs attached to a span and are not indexed for aggregation.

Naming convention: span name should be a low-cardinality operation identifier such as 'GET /orders/{id}' or 'SELECT orders', never include IDs or query parameters.

Span duration measures wall-clock time for the operation; self-time = span duration minus the sum of child span durations, which localizes where time is spent.

Instrumentation scope (formerly library/tracer name) plus version identifies which instrumentation produced a span, enabling per-library analysis.

Baggage carries arbitrary key-values across service boundaries independent of spans; it propagates to all downstream services and can leak sensitive data, so restrict it.

Span events should not replace logs for high-volume output; use them for discrete, meaningful moments (retry, cache miss, exception).

Full-stack observability correlates telemetry across the user (RUM/synthetic), application (APM, traces), infrastructure (hosts, containers, network), and dependencies (databases, queues, third parties).

Correlation keys: trace_id and span_id injected into logs, exemplars linking metrics to trace_ids, and shared resource attributes (service.name, host, pod) across all signals.

Exemplars attach a sampled trace_id to a metric data point, allowing a latency spike on a histogram to be clicked through to a representative trace.

RED method (rate, errors, duration) suits request-driven services; USE method (utilization, saturation, errors) suits resources such as CPU, memory, disks, and network links.

RUM captures real user sessions (page load, Core Web Vitals, JS errors); synthetic monitoring provides deterministic, baseline checks from fixed locations.

End-to-end latency decomposes into network, queueing, service, and dependency time; without spans at each hop, only the total is visible and root cause is ambiguous.

Dependency maps built from trace parent-child pairs reveal fan-out, retry storms, and chatty service-to-service calls that aggregate metrics hide.

Instrumentation coverage gap: services without telemetry appear as black boxes or missing edges in the service graph; track percentage of services emitting traces.

Shift-left observability: emit telemetry from CI/CD and test environments so failures are caught before production; build pipelines should fail on SLO-relevant regressions.

Security telemetry (audit logs, auth failures, egress volume, process execution) shares the pipeline with performance telemetry; correlation of a latency anomaly with an auth-failure spike is a common intrusion signal.

Supply-chain observability: track image digests, SBOMs, and provenance attestations as resource attributes so a vulnerable artifact can be traced to running services.

AIOps anomaly detection replaces static thresholds with learned baselines; it requires stable, labeled telemetry and produces false positives during topology or traffic changes.

Alert correlation and deduplication group related alerts into incidents; event correlation reduces alert noise but can mask a distinct root cause if grouping rules are too broad.

LLM/AI-assisted investigation depends on telemetry quality: it retrieves over traces, logs, and metrics, so missing context propagation degrades automated root-cause suggestions.

AI-generated code and agents increase service count and change rate, raising the value of automatic instrumentation over hand-written instrumentation.

Data governance: telemetry may contain PII (URLs, headers, payloads); redaction must happen in the Collector/agent before export to satisfy privacy requirements.

Mean time to detect (MTTD) and mean time to repair (MTTR) are the outcome metrics for observability investment; track them before and after changes.

Level 0: no telemetry beyond host checks. Level 1: per-service metrics and dashboards. Level 2: distributed tracing with context propagation. Level 3: correlated signals with SLOs and error budgets. Level 4: continuous, self-service, cost-governed observability.

SLO adoption is the strongest maturity marker: teams that define SLIs and error budgets can make release decisions from telemetry rather than opinion.

Instrumentation as code: telemetry configuration lives in version control alongside services, reviewed like application code, not clicked in a UI.

Ownership model: each service team owns its instrumentation, SLOs, and alerts; a central platform team owns the Collector fleet, standards, and cost controls.

Cost governance maturity: track cost per service and per signal, enforce retention tiers, and require justification for high-cardinality metrics.

Blameless incident review with telemetry evidence converts incidents into instrumentation improvements; repeat incidents with missing data indicate a maturity gap.

Onboarding time for a new service (time to first trace in production) is a measurable maturity indicator; targets are typically hours, not weeks.

Standardization: a shared semantic convention and resource-attribute schema across teams is a prerequisite for cross-service correlation and service graphs.

Container telemetry layers: host node, container runtime, pod (Kubernetes), and application; cAdvisor/kubelet expose container CPU, memory, and network counters.

Kubernetes resource attributes: k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.container.name, k8s.node.name; these join application traces to infrastructure metrics.

Pod churn means pod names are ephemeral; use stable identifiers (deployment, service, node) for dashboards and alerts, and pod UID for exact correlation.

Network observability sources: eBPF (kernel-level, no code changes), flow logs (VPC flow logs, Cilium Hubble), and service mesh telemetry (Envoy sidecar/proxy metrics and traces).

eBPF captures syscalls and socket-level data for L7 latency and connection errors without application instrumentation, but cannot see encrypted payload semantics unless TLS is terminated or uprobes are used.

DNS is a frequent hidden failure source: track DNS latency and NXDOMAIN/SERVFAIL rates; DNS timeouts often appear as generic upstream timeouts in application traces.

Container network symptoms: packet drops, retransmits, and connection resets point to CNI, MTU, or conntrack limits; check node-level conntrack table usage before blaming the app.

Service mesh sidecars add a hop; measure sidecar latency separately from application latency to avoid misattributing proxy overhead to business logic.

Resource limits cause OOMKills and CPU throttling (container_cpu_cfs_throttled_seconds_total); throttling raises tail latency without raising error rates, so alert on p99 and throttle ratio.