OpenTelemetry migration with tail sampling and sensitive attributes
A media company is replacing three proprietary monitoring agents with OpenTelemetry. It runs 300 services across three regions, emits 90 million spans and 1.2 TB of logs per day, and has 6,000 Kubernetes pods. Application teams own instrumentation; a central platform team owns collectors. Each region has a 12-node collector cluster, and the vendor backend charges by ingested span volume, not by sampled span. Current traces are head-sampled at 10% using a random sampler, so 95% of traces containing errors are discarded. The security team requires that HTTP Authorization headers and user email fields never leave the cluster. Two application teams already use custom attribute names such as user_id and http.method instead of OpenTelemetry semantic conventions. The migration must complete in 12 weeks, with no downtime and no increase in backend cost above 15%.
1. Recommend where to place tail-based sampling and how to preserve error traces without exceeding the cost constraint.
Read the worked answer
Tail sampling must see every span of a trace before it decides, so it cannot live in each application SDK and cannot sit behind a simple round-robin load balancer. The practical topology is a DaemonSet of OpenTelemetry Collector agents on each node forwarding to a regional gateway tier. The agents should use the load-balancing exporter keyed by trace ID so all spans for a trace reach the same gateway instance. The gateway runs the tail-sampling processor with a decision wait long enough to cover the slowest checkout path, plus a persistent queue for backend outages.
The policy should keep 100% of traces with status.code=ERROR, 100% above a latency threshold, and a 5-10% random baseline. At 90 million spans per day, 10% head sampling exports 9 million spans; replacing it with a 5% baseline plus all errors may be similar or lower in volume because errors are a small fraction. The team must measure the actual error-trace rate before promising the 15% ceiling. Gateways need memory sized for decision wait multiplied by span rate, and the agent tier needs CPU and disk buffers.
The trade-off is that tail sampling adds gateway complexity and latency in the pipeline, but it directly fixes the current defect: discarding 95% of error traces.
2. Propose controls for sensitive attributes and semantic-convention drift across 300 services.
Read the worked answer
Redaction must happen at the earliest trusted point, not only in the vendor backend. Put a Collector processor on the agent tier that deletes or hashes http.request.header.authorization and any attribute matching an email pattern; repeat a narrower processor at the gateway as defense in depth. Use the transform processor for regex-based email masking and an attributes processor for explicit deny-lists. Do not rely on application teams to remember every field, and do not send raw headers to a tail-sampling gateway where they could be logged during debugging.
Semantic-convention drift needs both mapping and prevention. The two teams using user_id should map to enduser.id, and http.method to the current convention http.request.method, at the Collector only as a temporary bridge. The durable fix is a shared instrumentation wrapper and a CI lint that rejects non-conventional attributes in code review. Publish a small internal schema covering required resource attributes, allowed span attributes, and cardinality limits. Track adoption by querying which services still emit legacy names. Also set the OpenTelemetry schema URL in resource attributes so backend transformations are versioned.
The trade-off is that Collector mapping hides drift rather than eliminating it; set a removal date for mappings and make the lint a merge gate.
3. Justify whether to deploy Collector agents on every node or only regional gateways for 6,000 pods, considering reliability and cost.
Read the worked answer
A gateway-only design is simpler to operate but weakens reliability and local context. Six thousand pods across 600 nodes can export directly to regional gateways, but an application pod then owns retry and buffering behavior, and a gateway outage can drop telemetry before it is batched. A DaemonSet agent per node provides local buffering, host and node metrics, and a stable egress point; it also reduces cross-node traffic and lets the platform team enforce redaction before data leaves the node.
The cost is roughly 600 agent pods plus CPU and memory, but agents can be small and the gateway tier can stay at 12 nodes per region for batching, tail sampling, and backend credentials. The key constraint is that tail sampling must be centralized by trace ID at the gateway, not at the agent, because an agent sees only spans from its own node. Agents should use persistent queues so a gateway restart does not lose data, and gateways should use the load-balancing exporter in front of the sampling tier.
A reasonable decision is agents plus gateways, with agents limited to forwarding and redaction, and gateways owning sampling and export. Gateway-only is acceptable only for low-volume services that can tolerate loss; it is not appropriate for the 90-million-span workload.