Observability Foundation flashcards

40 cards. Read the term and the definition here. The deck at the top of this page is optional.

Three Pillars

Metrics, logs, and traces. Metrics are aggregated numeric series, logs are discrete timestamped events, and traces are causal span trees following a request across service boundaries.

Observability vs monitoring

Monitoring answers predefined questions about known failure modes. Observability lets you ask new questions of high-cardinality, high-dimensional telemetry after deployment without shipping new code.

Error budget

One minus the SLO target, expressed as the allowed amount of unreliability over the window. A 99.9 percent SLO over 30 days permits about 43 minutes of bad events.

Cardinality explosion

Series count equals the product of label value counts. Unbounded labels such as user ID or request ID multiply series uncontrollably, so they belong in traces or logs instead.

Black-box vs white-box monitoring

Black-box observes externally visible symptoms like endpoint probes and synthetic checks. White-box observes internal state such as queue depth, cache hit ratio, and goroutine counts.

SLO burn rate

The rate at which the error budget is consumed relative to the SLO. A burn rate of 1 exhausts the budget exactly at the end of the window; higher values trigger faster alerts.

OpenTelemetry

A vendor-neutral CNCF project supplying APIs, SDKs, semantic conventions, and the OTLP protocol to generate and export traces, metrics, logs, baggage, and profiles.

OTLP ports

OpenTelemetry Protocol defaults: gRPC listens on port 4317 and HTTP with protobuf on port 4318. Both carry traces, metrics, and logs from SDKs or Collectors.

Resource attributes

Metadata describing the telemetry producer, such as service.name, service.version, host.name, and k8s.pod.name. service.name is required and defaults to unknown_service when unset.

W3C Trace Context

A standard header format, traceparent and tracestate, carrying trace ID, parent span ID, and sampling flags across service boundaries so distributed traces stay connected.

OpenTelemetry Collector

A standalone agent or gateway with receivers, processors, exporters, and connectors. It centralizes batching, memory limiting, sampling, redaction, and multi-backend fan-out.

SpanKind values

INTERNAL, SERVER, CLIENT, PRODUCER, and CONSUMER. Correct kinds are required for backends to infer service graphs and attribute latency to client versus server side.

Semantic conventions

Standardized attribute and span names defined by OpenTelemetry, such as http.request.method and db.system, so backends automatically interpret telemetry across languages and libraries.

Head-based sampling

A sampling decision made at the trace root before the trace completes. It is cheap and consistent when propagated, but cannot consider errors that occur later in the trace.

Tail-based sampling

A decision made after all spans of a trace arrive, usually in the Collector. It can retain all error or slow traces but requires buffering keyed by trace ID.

Critical path

The chain of spans whose exclusive durations sum to the end-to-end request latency. Optimizing spans outside this path does not reduce overall response time.

Broken context propagation

When trace headers are not forwarded across async boundaries, thread pools, or queues, downstream spans start new traces. Missing spans usually indicate this, not a backend fault.

Span links

A span field modeling causal relationships that are not parent-child, such as one consumer span linked to many producer spans in batched or asynchronous processing.

Clock skew in traces

Host clock differences make absolute span timestamps unreliable across machines. Use parent-child relationships and span durations, not raw timestamps, for causal latency analysis.

Span status

A field with values Unset, Ok, or Error. Set Error only for failed operations and record the exception as a span event with type, message, and stacktrace.

Span attributes vs events

Attributes are indexed key-values used for querying and aggregation. Events are timestamped annotations on a span, useful for discrete moments like retries or exceptions.

Span self-time

A span's duration minus the sum of its child span durations. It localizes where time is spent inside an operation rather than in downstream calls.

Span naming rule

Use a low-cardinality operation identifier such as GET /orders/{id}. Never embed request IDs, user IDs, or query parameters, because they explode unique span names.

Baggage

Arbitrary key-value pairs propagated across service boundaries independently of spans. It reaches every downstream service, so sensitive data must be excluded.

Exemplars

Sampled trace IDs attached to individual metric data points. They let an operator jump from a latency histogram spike directly to a representative trace.

RED method

Rate, errors, and duration. It is the standard metric set for request-driven services and maps naturally to spans and service-level dashboards.

USE method

Utilization, saturation, and errors. It applies to resources such as CPU, memory, disks, and network links rather than to request-handling services.

Full-stack correlation keys

trace_id and span_id injected into logs, exemplars linking metrics to traces, and shared resource attributes like service.name and pod name across all signals.

Shift-left observability

Emitting and validating telemetry in CI and test environments so regressions in latency, errors, or resource use are caught before production deployment.

Security telemetry correlation

Combining audit logs, auth failures, egress volume, and process execution with performance signals. A latency anomaly coinciding with auth-failure spikes often signals intrusion.

AIOps anomaly detection

Learned baselines replacing static thresholds. It needs stable, labeled telemetry and produces false positives during topology changes or traffic shifts.

Telemetry PII redaction

Removing sensitive values from URLs, headers, and payloads in the agent or Collector before export, so privacy requirements are met before data reaches any backend.

Observability maturity levels

A progression from host checks to per-service metrics, then distributed tracing, then correlated signals with SLOs, then self-service cost-governed observability.

Instrumentation as code

Telemetry configuration stored in version control and reviewed like application code, rather than configured manually in a vendor UI, enabling reproducibility and audit.

Service onboarding time

The elapsed time from a new service's creation to its first production trace. It is a concrete maturity indicator; mature organizations measure it in hours.

eBPF observability

Kernel-level instrumentation capturing syscalls and socket data for L7 latency and connection errors without application code changes, though encrypted payloads remain opaque.

Kubernetes resource attributes

k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.container.name, and k8s.node.name. They join application traces to infrastructure metrics and logs.

CPU throttling signal

container_cpu_cfs_throttled_seconds_total rising indicates CPU limits are too low. It raises tail latency, especially p99, without necessarily raising error rates.

DNS as hidden failure

DNS latency and NXDOMAIN or SERVFAIL rates are frequent root causes. DNS timeouts typically surface in application traces as generic upstream connection timeouts.

Sidecar latency attribution

Service mesh proxies add a network hop. Measuring proxy latency separately from application span duration prevents misattributing proxy overhead to business logic.