Packet loss invisible to node and mesh dashboards
A financial services firm runs 4,000 pods on 600 nodes across three Kubernetes clusters. The security team mandates mTLS between all workloads and requires Layer 7 network policy. The observability team uses a sidecar-based service mesh, which adds 12 ms p95 latency to each call and consumes 0.3 vCPU per pod. During a recent incident, checkout pods on one node had 8-12% packet loss; application metrics showed normal CPU and memory, node metrics showed normal load, and the mesh dashboard showed no errors. The network team found a misconfigured TC qdisc on the node's primary interface causing drops for a specific pod CIDR. The firm is evaluating an eBPF-based approach to replace sidecars for observability and security. It has a 6-month window, a strict change-control process, and 40% of workloads still run on kernels older than 5.4.
1. Assess whether eBPF can replace sidecars for mTLS and Layer 7 policy, and what blind spots remain.
Read the worked answer
eBPF is strong for network observability: socket-level flows, TCP retransmits, DNS latency, connection failures, and L3/L4 policy. It is not a drop-in replacement for sidecar mTLS and Layer 7 policy in this environment. If mTLS terminates in the sidecar, eBPF on the host sees encrypted bytes; it cannot inspect HTTP/2 or gRPC semantics without complex uprobes into TLS libraries, and those probes are kernel- and library-version dependent. Workload identity, certificate rotation, and L7 authorization are mature in the sidecar model and would need a separate control plane if sidecars are removed.
The safe position is hybrid: keep sidecars for mTLS termination and L7 policy, use eBPF for network flow visibility, DNS, retransmits, and node/pod-level packet drops. eBPF can also provide a lower-overhead data path for observability without removing the security boundary. Blind spots remain for encrypted payload contents, application-level retries, queue wait, and business context; those still require application instrumentation and mesh or proxy telemetry.
The kernel constraint is decisive: 40% of workloads are below 5.4, where BTF/CO-RE and some tracing features are limited. A universal replacement in six months is not credible.
2. Using the incident, explain why node and application metrics missed the packet loss and what telemetry to add.
Read the worked answer
Node CPU, memory, and load measure resource pressure, not network health. A TC qdisc dropping 8-12% of packets for one pod CIDR may barely move aggregate node load, and application metrics can look normal because TCP retransmits hide loss as added latency rather than errors. The mesh dashboard showed no errors likely because retries succeeded and the mesh measures request outcomes, not per-packet drops; it may also aggregate across all pods and dilute one node's CIDR-specific problem. Averages over 600 nodes hide a single-node fault.
The team should add pod-level and flow-level network telemetry: TCP retransmit rate, retransmit timeout, RTT, connection resets, DNS latency and failure rate, and qdisc drop counters from the host. eBPF flow logs can attribute drops to source and destination pods and to the specific interface, which is what the TC misconfiguration required. Add mesh RED metrics with retry counts and distinguish original requests from retries. Correlate network latency with trace spans to see which calls slowed. Alert on retransmit rate and packet drops per pod CIDR, not just node averages. Synthetic probes between checkout and its dependencies would also have caught the loss before customers did.
3. Recommend a migration path given kernel constraints and strict change control.
Read the worked answer
The path should be phased and evidence-driven, not a six-month sidecar removal. First, inventory kernel versions and identify the 60% of nodes on 5.4 or later with BTF support. Deploy eBPF observability in canary mode on a small set of those nodes, starting with flow logs, TCP retransmits, DNS, and drop attribution. Run it alongside the mesh and compare findings against the recent incident. Define success criteria: coverage of pod-to-pod flows, overhead under an agreed CPU budget, detection of a seeded packet-loss fault, and no increase in MTTR.
In parallel, upgrade kernels for the remaining 40% through the normal change process, because eBPF security features cannot be uniform without them. Keep sidecars for mTLS and L7 policy throughout the six months; do not couple observability migration to security replacement. If eBPF observability proves reliable, phase it to more clusters and use it to reduce sidecar telemetry load, not to remove the security function.
Change control should be satisfied with feature flags, canary namespaces, rollback plans, and a dual-run period. A six-month window is enough for observability, not for replacing mTLS and L7 enforcement across 4,000 pods.