Streaming Telemetry Shows Gaps Under Load
You have configured gNMI streaming telemetry from 40 leaf switches to a collector, subscribing to interface counters at a 10-second interval. During peak hours, the collector shows gaps: some devices report every 10s, others every 30-60s, and a few stop for minutes. Interface counters are the only subscription. The collector is a single VM with 8 vCPU and 16 GB RAM. The management network is 1 Gbps shared with eAPI polling from a separate monitoring system that polls every 60s.
1. What are the likely causes of the telemetry gaps, and how would you isolate them?
Read the worked answer
Likely causes, in order of probability:
- Collector overload - 40 devices × 10s × many interfaces can be a high message rate; a single 8-vCPU VM may fall behind, causing buffering and gaps. Check collector CPU, memory, and queue depth.
- Management network contention - the 1 Gbps link is shared with eAPI polling; telemetry bursts plus polling can cause drops. Check interface utilization and drops on the mgmt link.
- Device-side subscription limits - some EOS versions limit concurrent gNMI subscriptions or sampling rates; devices that "stop for minutes" may be hitting a limit or a session timeout.
- TCP session resets - gNMI is TCP; a reset drops the subscription until reconnect. Check collector logs for reconnect events per device.
Isolation: correlate gaps with collector CPU spikes, mgmt link utilization, and per-device reconnect logs. If gaps align with collector CPU, it's the collector; if they align with network bursts, it's the network; if specific devices only, it's device-side.
2. Propose a remediation plan that keeps the 10-second interval where it matters.
Read the worked answer
- Reduce message volume: subscribe only to interfaces of interest (uplinks, not all ports) and use on-change or a longer interval for low-value counters. Keep 10s only for critical uplinks.
- Scale the collector: move to a larger VM or split collectors (e.g., 20 devices each) to halve per-collector load.
- Separate the management path: move telemetry to a dedicated VLAN/interface or QoS-prioritize it above eAPI polling; reduce eAPI poll frequency or move it off-peak.
- Tune device subscriptions: verify per-device subscription limits and set sane
sample-interval; avoid duplicate subscriptions. - Add collector-side buffering and backpressure with disk spooling so short bursts don't create gaps, and alert on subscription reconnect rate.
- Validate by watching gap rate, collector CPU, and mgmt utilization for a week before declaring success.