Telemetry Gap: Why Dashboards Miss a Silent Degradation
Operations relies on a CX 10000 telemetry pipeline exporting flow and interface counters to an analytics platform. A gradual degradation (rising retransmits and tail latency on app->db traffic) went unnoticed for two days; dashboards showed 'green' because they chart average throughput and link up/down only. The DPU flow exports are sampled at a low rate and the collector retains 24 hours. The team wants to add more dashboards but keep the same sampling and retention.
1. Why did the existing telemetry miss the degradation, and what should change?
Read the worked answer
The dashboards measure the wrong signals at the wrong resolution:
- Averages hide tails: average throughput stays flat while tail latency and retransmits climb. Need p95/p99 latency, TCP retransmit rate, and per-flow RTT - not averages.
- Sampling too coarse: low-rate flow sampling under-samples the specific app->db flow mix; raise sampling for the critical policy/segment or use targeted flow export.
- Retention too short: 24 hours cannot show a two-day trend; extend retention for key counters so slow drifts are visible.
- Missing link between flow and health: correlate DPU flow exports with interface error/retransmit counters and firewall session metrics.
Add more dashboards only after fixing the signals: chart retransmits, p99 latency, session-table utilization, and deny/allow ratios per tier, with alerting thresholds on rate-of-change - not just static averages.
2. Design an alert that would have caught this in hours, not days.
Read the worked answer
A rate-of-change alert on leading indicators, not a static threshold:
- Metric: TCP retransmit rate for the app->db flow group (from DPU flow export) and p99 RTT.
- Condition: retransmit rate exceeds baseline by a defined multiple (e.g., 3x the 7-day median) sustained for 15 minutes, OR p99 RTT drifts upward across consecutive windows.
- Correlation: require the same window to show no link down/up (rules out simple outages) and stable session counts (rules out capacity).
- Routing: page the on-call with the top talkers and affected flow tuples attached.
Key design points: baseline-relative (catches slow drift), sustained-window (avoids noise), and tied to the specific tier pair. Validate by replaying the two-day incident data to confirm the alert fires within hours.