Observability Gap After a Microburst
A financial exchange runs Arista switches with streaming telemetry to CloudVision and a separate gRPC collector. A trading application reports intermittent 200-400 ms stalls during market open. Interface counters show no errors and average utilization is 40%. The team has 1-minute SNMP polling and 10-second gRPC interface counters. A packet capture on the server shows retransmissions. The network team suspects microbursts on a 100G uplink shared by several trading hosts. The compliance team requires evidence retention for 90 days. The team wants to know what telemetry to add, how to correlate it, and whether CloudVision alone is sufficient.
1. Given 1-minute SNMP and 10-second gRPC counters, why is the team likely missing the stall, and what telemetry would you add?
Read the worked answer
A 200-400 ms stall can be caused by a microburst that is invisible at 10-second granularity: a 10-second average can look like 40% utilization while a sub-second burst saturates the 100G uplink and fills the egress queue. SNMP at 1 minute is even less useful.
Add:
- High-frequency interface and queue counters via gRPC streaming at sub-second intervals (for example, 100 ms or faster) on the suspect uplink and on the trading host ports.
- Buffer/queue depth and drop counters per queue, including PFC and pause frames if lossless is configured.
- INT or in-band telemetry if supported, to see per-hop latency and queue occupancy for the trading flows.
- Timestamped packet capture or flow telemetry on the server and the leaf to correlate retransmissions with queue events.
The judgement is that the existing polling interval is the wrong tool for the symptom; add sub-second queue telemetry before changing the network.
2. How would you correlate the telemetry with the application stalls, and what would confirm the microburst hypothesis?
Read the worked answer
Correlate on a common time base. Ensure all collectors use NTP/PTP and that timestamps are comparable across CloudVision, the gRPC collector, and the server capture. Then align the 200-400 ms stall windows with:
- Queue depth spikes and drop counters on the uplink.
- PFC pause or ECN marks on the trading flows.
- Server retransmission timestamps.
Confirmation: if the stall windows coincide with sub-second queue occupancy spikes or drops on the shared 100G uplink, and the server retransmits in the same window, the microburst hypothesis is supported. If queue depth is flat and no drops occur, look elsewhere (for example, host NIC, application, or a control-plane event).
The judgement is to require temporal coincidence, not just correlation of averages, before concluding.
3. The compliance team requires 90-day retention. Is CloudVision alone sufficient, and what architecture would you recommend?
Read the worked answer
CloudVision is a management and visibility platform, not a long-term telemetry archive at sub-second granularity. Retaining 90 days of high-frequency queue and interface telemetry across many switches in CloudVision alone is not the right fit and may exceed its intended retention and scaling.
Recommend a tiered architecture:
- CloudVision for device management, topology, and near-term visibility.
- A dedicated gRPC streaming collector plus time-series database (for example, a TSDB) for sub-second telemetry, with retention sized for 90 days.
- Object storage for long-term retention and compliance export.
Validate retention and query performance against the 90-day requirement, and document the data flow so compliance can audit it. The judgement is to separate management visibility from long-term telemetry retention rather than forcing CloudVision to be both.