Deployment, Operations, and Troubleshooting
Operational success depends on configuration hygiene, resource limits, and knowing where telemetry can silently drop.
Deployment: choose agent vs. gateway topologies; run the collector with adequate CPU/memory; use the memory_limiter processor to prevent OOM and the batch processor to reduce export overhead. Configure retry and queue settings on exporters to survive backend outages. Understand sampling: head sampling (decided at trace start, e.g., parent-based, trace-ID-ratio) vs. tail sampling (decided after spans complete, in the collector, can keep all errors/slow traces). Tail sampling requires all spans of a trace to reach the same collector instance - hence load-balancing by trace ID.
Troubleshooting playbook: (1) Is the SDK exporting? Check exporter endpoint, protocol, TLS, and auth. (2) Is the collector receiving? Check receiver config and ports. (3) Is the pipeline wired? Every component declared and referenced. (4) Are processors dropping data? Filter processors and memory_limiter under pressure can drop. (5) Is sampling discarding traces? (6) Are propagators consistent? Common symptoms: no data at all (endpoint/auth/wiring), partial traces (propagation or sampling), missing attributes (wrong semantic conventions or dropped), collector crashes (missing memory_limiter), export failures (backend down, no retry/queue).
Traps: assuming the SDK retries forever; forgetting that tail sampling needs trace-affinity; enabling debug logging in production carelessly; ignoring collector self-telemetry.
Rule of thumb: troubleshoot in the data path order - SDK → network → receiver → processor → exporter → backend.
Worked example: Traces arrive but ~half are incomplete. Likely cause: tail sampling spread across collectors without trace-ID routing, or inconsistent propagators. Verify load balancing key and propagator config.