Observability Gap in Multi-Cloud Kubernetes
A retail company runs 2,000 pods across EKS in us-west-2 and AKS in West Europe. The SLO for checkout is p99 under 300 ms, but dashboards show 15% TCP retransmits on node-to-node traffic and intermittent 2-second tail spikes every 10 minutes. The platform team has Prometheus metrics from kube-state-metrics and node_exporter, plus cloud provider flow logs. However, service mesh metrics are sampled at 1%, and flow logs are aggregated at 10-minute intervals with 7-day retention. A recent incident took 45 minutes to diagnose because the team could not tell whether latency originated in the application, the CNI overlay, or cross-cloud egress. The company pays $18,000 per month for observability and wants to cap additional spend at 15%. There are 300 services and 12,000 flows per second at peak. The team needs faster mean time to detect (MTTD) and root cause without collecting full packet captures continuously.
1. Identify the most important telemetry gaps and propose a collection design that stays within the 15% budget increase.
Read the worked answer
The main gaps are tail visibility and correlation. One-percent mesh sampling will miss 2-second spikes, and 10-minute flow logs cannot show whether retransmits are between pods, nodes, or clouds. Add eBPF-based network telemetry at the node level to capture TCP retransmits, RTT, and connection resets without full packet capture. Increase service mesh sampling to 10% for normal traffic but use tail-based sampling to keep 100% of traces with errors or latency above 250 ms. Change VPC flow logs to 1-minute aggregation and 30-day retention in hot storage, then tier to object storage. Add OpenTelemetry tracing with 5% head sampling and always sample checkout errors. To stay within budget, reduce unused metrics, aggregate node metrics, and use open-source collectors. Do not collect full packet captures continuously; trigger them on alert.
2. Design alerts for p99 tail latency and retransmits that will reduce MTTD without causing alert fatigue.
Read the worked answer
Use SLO burn-rate alerts rather than static thresholds. For the 300 ms p99 SLO, alert when the burn rate over 5 minutes and 1 hour exceeds the error budget policy, and page only for fast burn. Add a retransmit alert when the pod or node retransmit rate exceeds 2% for 5 minutes and is at least 10% above its own weekly baseline. Correlate with deployment events, node pressure, and cross-cloud egress. Create a separate warning for p99 above 250 ms for 10 minutes. Group alerts by service and region, and inhibit lower-level alerts when a higher-level incident is open. Include links to the trace query, flow logs, and the topology map. Review alert precision weekly and tune thresholds. This reduces noise and focuses the team on customer impact.
3. How would you determine whether the 2-second spikes originate in the CNI overlay, the application, or cross-cloud egress?
Read the worked answer
Use a layered correlation approach. First compare eBPF retransmit and RTT metrics at the pod, node, and cross-node level; if retransmits are high only between nodes on one CNI, suspect the overlay or MTU. Check CNI metrics such as drops, encap errors, and tunnel interface counters. Second, inspect distributed traces: if the server span is fast but the client span is slow, the delay is in the network or client; if the server span is slow, it is the application. Third, compare same-cloud versus cross-cloud calls. For cross-cloud, check the SD-WAN or TGW/VPN path, egress NAT, and internet weather. Use on-demand packet captures at the source and destination during an alert, and look for retransmission timeouts, duplicate ACKs, and MTU-related drops. A topology map that joins traces to flows will show the offending hop.