I85PREP
⌘ K
OTCA study materials
OTCA study resources

The Collector That Drops Data Under Load and Hides It

OTCA · Put your knowledge to work in context.

All case studies

The Collector That Drops Data Under Load and Hides It

A collector gateway is configured with memory_limiter (limit 80% of 2Gi, spike 20%), batch (defaults), and otlp receivers/exporters. During a traffic spike (3× normal, ~50k spans/sec), the backend receives roughly 60% of expected spans. The team sees no errors in application logs, and the collector's own logs are quiet. otelcol_receiver_refused_spans is non-zero but nobody alerts on it. The batch processor is placed before memory_limiter in the pipeline, and the exporter has no retry/queue configuration (defaults). The team's on-call runbook says 'if data is missing, check the app' - which found nothing.

1. Trace where the 40% of spans are being lost. Identify each loss point in the pipeline and the specific metric or signal that would reveal it.

Read the worked answer

Loss point 1 - receiver backpressure (likely the largest). The memory_limiter refuses data when memory exceeds the limit. Because memory_limiter is placed after batch (wrong order), it is reacting to already-batched memory, and the refusal happens at the receiver boundary. otelcol_receiver_refused_spans counts these. At 3× load, this is the primary 40%.

Loss point 2 - exporter queue overflow. With no explicit sending_queue, the exporter uses defaults (queue enabled but small, e.g. 1000 batches). Under sustained 50k spans/sec, if the backend is slower than ingest, the queue fills and the exporter drops with otelcol_exporter_enqueue_failed_spans (and otelcol_exporter_sent_spans lagging). Defaults also mean a bounded retry; once the queue is full, spans are dropped.

Loss point 3 - batch timeout/size mismatch. Default batch (8192 spans or 200ms) is fine in steady state but under spike the 200ms timeout can hold data while memory climbs, worsening limiter refusals.

Signals to check:

  • otelcol_receiver_refused_spans (receiver, by transport) - the smoking gun for limiter refusals.
  • otelcol_exporter_enqueue_failed_spans - queue full.
  • otelcol_exporter_sent_spans vs otelcol_receiver_accepted_spans - the gap is the loss.
  • otelcol_processor_batch_batch_send_size and otelcol_process_rss - memory pressure.
  • otelcol_exporter_send_failed_spans - backend errors (if retry exhausted).

Why app logs are quiet: the loss is entirely in the collector; apps successfully sent (or were refused at the network edge, which many SDKs log only at debug). The runbook's 'check the app' is the wrong first step - check the collector's own telemetry first.

2. Give the corrected pipeline ordering and exporter configuration, and explain the ordering rationale. Then state what to alert on so this is caught next time.

Read the worked answer

Corrected pipeline order (processors run in listed order):

  1. memory_limiter - first, so it sheds load before any buffering/processing, protecting the process from OOM and refusing at the receiver with a clear metric.
  2. (optional) attributes/filter/redaction - cheap, before batching.
  3. batch - last before export, so batching operates on already-accepted data and doesn't inflate memory ahead of the limiter.

Exporter config (explicit, not defaults):

  • sending_queue: enabled: true, queue_size sized for the spike (e.g. 5000–10000 batches), num_consumers > 1.
  • retry_on_failure: enabled: true, initial_interval, max_interval, max_elapsed_time tuned so transient backend slowness doesn't drop data. Note: infinite retry risks memory growth - pair with memory_limiter.
  • Consider timeout per export.

Ordering rationale: memory_limiter must see incoming data volume and protect the process; if batch runs first, the limiter reacts to buffered memory (delayed) and refusals happen unpredictably. Batching last maximizes efficiency without hiding memory pressure.

Alerting (the real fix for 'nobody noticed'):

  • rate(otelcol_receiver_refused_spans[5m]) > 0 - page.
  • rate(otelcol_exporter_enqueue_failed_spans[5m]) > 0 - page.
  • Ratio sent / accepted < 0.99 over 5m - warn.
  • otelcol_process_rss > 80% of limit - warn.
  • otelcol_exporter_send_failed_spans sustained - page.

Runbook change: first step is 'check collector self-telemetry (refused/enqueue_failed/sent-vs-accepted)', not 'check the app'. Also add a load test at 3× to validate the new queue sizing.