OTCA cheat sheet

OpenTelemetry is a vendor-neutral CNCF project providing APIs, SDKs, and tooling to generate, collect, and export telemetry; it does not include a backend or storage system.

Three signals are core: traces (request paths), metrics (aggregated numeric measurements), and logs (timestamped event records); baggage is contextual metadata, not a signal.

API is a no-op by default; behavior is supplied by an SDK implementation registered at runtime, so library code can depend on the API alone.

A span represents a unit of work with a span context (trace ID, span ID, flags), name, timestamps, attributes, events, links, and status.

Instrumentation scope (formerly library) identifies the instrumenting code via name, version, and optional attributes attached to emitted telemetry.

Resource describes the entity producing telemetry (service.name, host, process, cloud) and is merged across providers; service.name is strongly recommended.

Trace context is propagated via W3C Trace Context (traceparent/tracestate); baggage uses the W3C Baggage header.

Semantic conventions define standard attribute names and values (e.g., http.request.method, db.system) so telemetry is portable across tools.

OpenTelemetry is a merger of OpenTracing and OpenCensus; its specification, APIs, SDKs, and Collector evolve under separate stability guarantees.

A trace is a DAG of spans sharing one trace ID; the root span has no parent, and parent_span_id links child spans to their caller.

Span status is Unset, Ok, or Error; Error should be set for failed operations and is distinct from recording an exception as a span event.

Sampling decisions: head sampling occurs in the SDK before export; tail sampling occurs in the Collector after spans are observed.

Metric instruments are Counter (monotonic sum), UpDownCounter (non-monotonic sum), Histogram (distribution with buckets), and Gauge (current value).

Temporality: delta reports changes since last collection; cumulative reports totals since start. Prometheus typically expects cumulative; OTLP supports both.

Logs have a severity number and text, a body, attributes, and optionally a trace ID and span ID to correlate with traces.

Baggage is a set of key-value pairs propagated across service boundaries; it is not exported as telemetry automatically and must be read and recorded explicitly.

Exemplars attach a sampled trace reference (trace ID, span ID) to metric data points, linking metrics back to individual traces.

Cardinality of metric attributes (dimensions) drives cost and memory; high-cardinality values like user IDs should not be metric attributes.

Zero-code (automatic) instrumentation uses agents, JVM agents, or bytecode/eBPF instrumentation to create spans without source changes.

Manual instrumentation uses the OpenTelemetry API: obtain a Tracer, start spans, set attributes, and end spans within a scope.

A TracerProvider is configured once per process; Tracer instances are created from it and are cheap, long-lived handles.

A MeterProvider creates Meters; instruments are created from a Meter and should be created once at initialization, not per request.

Span lifecycle: start span, make it current in context, do work, set status and attributes, then end it; ending a span records its end timestamp.

Context must be propagated through in-process calls using the context API (e.g., context.with(span)) so child spans attach to the correct parent.

Auto-instrumentation libraries can be combined with manual spans; the manual span becomes a child of the active auto-instrumented span.

Span links associate a span with causally related spans (e.g., batch processing) without making it a parent, avoiding distorted trace trees.

SDK configuration includes sampler, span processors, exporters, resource, and limits such as attribute count and span event count.

The Collector pipeline has receivers (ingest), processors (transform/filter), exporters (send), and connectors (join pipelines); pipelines are defined per signal.

Receivers include OTLP (gRPC and HTTP), and protocol-specific ones like Prometheus, Jaeger, Zipkin, and filelog.

Processors include batch, memory_limiter, attributes, resource, filter, and tail_sampling; ordering matters (e.g., memory_limiter before batch).

Exporters include otlp, otlphttp, debug, logging, prometheus, and vendor-specific exporters.

Configuration is YAML with top-level sections: receivers, processors, exporters, connectors, extensions, service.pipelines, and service.telemetry.

The batch processor groups telemetry by size or timeout; the memory_limiter processor refuses data when memory exceeds soft/hard limits to prevent OOM.

Extensions (e.g., health_check, pprof, zpages) run alongside pipelines and are enabled under service.extensions.

A deployment mode is agent (per-host, typically with batch and forwarding) or gateway (centralized, with sampling and enrichment).

The Collector can run as a standalone binary or in a container; configuration is passed with --config and can be composed from multiple files.

W3C Trace Context traceparent format: version-traceid-parentid-traceflags, e.g., 00-<32 hex>-<16 hex>-01 where 01 means sampled.

tracestate carries vendor-specific key-value pairs; it must be forwarded unchanged unless the vendor owns the entry.

Propagators are configured on the SDK; composite propagators combine trace context and baggage (e.g., tracecontext,baggage).

Baggage header format is key=value pairs separated by commas; values are URL-encoded and should be treated as untrusted input.

In-process propagation uses a context object; out-of-process propagation uses inject/extract on carriers such as HTTP headers or message attributes.

Semantic conventions for HTTP include http.request.method, url.full, http.response.status_code, and server.address.

RPC conventions include rpc.system, rpc.service, rpc.method; database conventions include db.system, db.operation, db.namespace.

Messaging conventions include messaging.system, messaging.destination.name, and messaging.operation (send, receive, process).

Span kind (server, client, producer, consumer, internal) reflects the role of the span and affects how context is propagated.

OTLP is the native protocol over gRPC (default port 4317) and HTTP/protobuf (default port 4318); most SDKs support both.

Environment variables configure SDKs: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES.

Sampling trade-off: head sampling reduces cost early but may miss errors; tail sampling in the Collector can keep errors and slow traces.

Common failure: exporter cannot reach the Collector - check endpoint, TLS, and DNS; the SDK logs export errors internally.

Missing spans across services usually indicate propagation failure: verify propagators are configured and headers are forwarded by proxies.

Collector debugging: enable the debug/logging exporter, check service.telemetry logs, and use zpages or health_check extensions.

Resource and attribute limits (OTEL_ATTRIBUTE_COUNT_LIMIT, span event limits) prevent unbounded memory growth from high-cardinality data.

Backpressure: the SDK uses a batching span processor with a queue; a full queue drops spans, so size queues and Collector capacity together.

Rollout strategy: deploy the Collector as a DaemonSet (agent) and a Deployment (gateway); scale gateways horizontally and use load balancing for OTLP gRPC.

OTLP is the native protocol over gRPC (default port 4317) and HTTP/protobuf (default port 4318); most SDKs support both.

Environment variables configure SDKs: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES.

Sampling trade-off: head sampling reduces cost early but may miss errors; tail sampling in the Collector can keep errors and slow traces.

Common failure: exporter cannot reach the Collector - check endpoint, TLS, and DNS; the SDK logs export errors internally.

Missing spans across services usually indicate propagation failure: verify propagators are configured and headers are forwarded by proxies.

Collector debugging: enable the debug/logging exporter, check service.telemetry logs, and use zpages or health_check extensions.

Resource and attribute limits (OTEL_ATTRIBUTE_COUNT_LIMIT, span event limits) prevent unbounded memory growth from high-cardinality data.

Backpressure: the SDK uses a batching span processor with a queue; a full queue drops spans, so size queues and Collector capacity together.

Rollout strategy: deploy the Collector as a DaemonSet (agent) and a Deployment (gateway); scale gateways horizontally and use load balancing for OTLP gRPC.

OTLP is the native protocol over gRPC (default port 4317) and HTTP/protobuf (default port 4318); most SDKs support both.

Environment variables configure SDKs: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES.

Sampling trade-off: head sampling reduces cost early but may miss errors; tail sampling in the Collector can keep errors and slow traces.

Common failure: exporter cannot reach the Collector - check endpoint, TLS, and DNS; the SDK logs export errors internally.

Missing spans across services usually indicate propagation failure: verify propagators are configured and headers are forwarded by proxies.

Collector debugging: enable the debug/logging exporter, check service.telemetry logs, and use zpages or health_check extensions.

Resource and attribute limits (OTEL_ATTRIBUTE_COUNT_LIMIT, span event limits) prevent unbounded memory growth from high-cardinality data.

Backpressure: the SDK uses a batching span processor with a queue; a full queue drops spans, so size queues and Collector capacity together.

Rollout strategy: deploy the Collector as a DaemonSet (agent) and a Deployment (gateway); scale gateways horizontally and use load balancing for OTLP gRPC.

OTLP is the native protocol over gRPC (default port 4317) and HTTP/protobuf (default port 4318); most SDKs support both.

Environment variables configure SDKs: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES.

Sampling trade-off: head sampling reduces cost early but may miss errors; tail sampling in the Collector can keep errors and slow traces.

Common failure: exporter cannot reach the Collector - check endpoint, TLS, and DNS; the SDK logs export errors internally.

Missing spans across services usually indicate propagation failure: verify propagators are configured and headers are forwarded by proxies.

Collector debugging: enable the debug/logging exporter, check service.telemetry logs, and use zpages or health_check extensions.

Resource and attribute limits (OTEL_ATTRIBUTE_COUNT_LIMIT, span event limits) prevent unbounded memory growth from high-cardinality data.

Backpressure: the SDK uses a batching span processor with a queue; a full queue drops spans, so size queues and Collector capacity together.

Rollout strategy: deploy the Collector as a DaemonSet (agent) and a Deployment (gateway); scale gateways horizontally and use load balancing for OTLP gRPC.