Intermittent 10 a.m. Loss in a 400G VXLAN Fabric
A financial exchange runs a 25G/400G VXLAN fabric with 96 leaf switches and eight spines. Users report 2% packet loss and latency spikes between 10:00 and 11:00 daily, coinciding with backup and analytics jobs. The monitoring stack polls SNMP every five minutes and collects sFlow at 1:512 sampling; it shows no link saturation, but spine egress buffer drop counters increase during the window. ECN marking is enabled on leaf-to-spine links, and PFC is enabled for RoCE traffic. The NOC has not found a single failing link. A recent change added 20 analytics nodes with 100G NICs and jumbo frames. The exchange cannot accept packet capture on production links and needs a data-driven troubleshooting plan. The fabric carries trading and market-data traffic, so a 2% loss is material and any maintenance window is tightly controlled.
1. Formulate a hypothesis-driven troubleshooting plan to isolate the 10 a.m. loss, including what telemetry you would add.
Read the worked answer
Start with the time correlation but do not assume backups are the cause. The five-minute SNMP poll cannot see a 30-second microburst, so the first change is telemetry: streaming queue-depth and drop counters at 1-second or 100-millisecond intervals, plus ECN and PFC counters per queue. Build a hypothesis list: incast from 20 new 100G analytics nodes causing spine egress microbursts; jumbo-frame MTU mismatch causing drops on one path; PFC pause propagation from RoCE traffic creating head-of-line blocking for other traffic; and analytics and backup sharing a queue with production. Test by correlating per-spine queue drops with the analytics job start time, checking ECN marks versus drops, and verifying MTU on every hop. Since captures are not allowed, use INT or streaming telemetry and switch-based flow export. Isolate one analytics node and one backup job in a maintenance window to reproduce.
2. Interpret the ECN and PFC counters and decide whether to tune queues, redesign traffic classes, or add uplink capacity.
Read the worked answer
Read the counters as a story. ECN markings mean the fabric is signaling congestion before drops; if ECN is working, production should back off, and sustained drops suggest either a queue that ignores ECN or a burst too fast for the marking threshold. PFC pauses are more dangerous: a pause on a lossless queue can block all traffic on that port, including production, if the queues are not separated. Spine egress buffer drops during the 10:00 window point to oversubscription or incast at the spine, not a failing link. The right fix is not a bigger buffer; it is QoS separation and traffic engineering. Put analytics and backup in a scavenger or low-priority queue with a capped share, keep RoCE in a dedicated lossless queue with PFC, and set ECN thresholds so production marks before the queue fills. Then re-measure. If drops persist at the spine with all classes separated, the spine uplink is undersized for the new 100G nodes.
3. Propose a monitoring architecture that would catch this class of problem before users report it.
Read the worked answer
Move from polling to a telemetry-first architecture. Keep SNMP for inventory and coarse health, but add streaming telemetry for interface queue depth, drop reasons, ECN/PFC, and buffer utilization at sub-second resolution; add flow telemetry or sFlow with adaptive sampling for elephant flows; and synthetic probes between leaf pairs for latency and loss. Alert on p99 latency and queue depth, not just link utilization, because a 40% utilized link can still microburst. Correlate with job schedulers and CMDB so a backup start is an annotation, not a mystery. Retain high-resolution data for at least 30 days for the monthly backup cycle. Finally, run a quarterly what-if: add a rack of 100G nodes in a model and check spine oversubscription before the change, not after the 10 a.m. loss.