Telemetry at 10-Second Resolution for 1,200 Switches
You operate 1,200 Cisco Nexus 9000 switches running NX-OS 10.2(5). Capacity planning needs interface in/out octets and discards every 10 seconds for 48 hours, about 1.2 million time series. The current monitoring stack polls 300 SNMP OIDs per switch every 5 minutes with 24 pollers; SNMP walks already take 70 seconds per switch and the collector cluster is at 80% CPU. A 10-second SNMP interval would require 24x the polling capacity and has caused CPU spikes on leaf switches during previous attempts. The network team has gNMI-capable collectors and can use either dial-in or dial-out. Security requires TLS and per-switch authentication, and the collectors are behind a load balancer. The WAN between sites is 1 Gbps with 40 ms RTT. You need to choose a telemetry design and justify it to a change board.
1. Compare SNMP polling with model-driven telemetry for this requirement. Use the numbers in the setup to explain why 10-second SNMP is not viable.
Read the worked answer
SNMP is a pull model with per-OID request overhead and agent CPU cost. Today: 1,200 switches × 300 OIDs / 300 s = 1,200 OID polls per second, and a 70-second walk per switch. At 10 seconds the same math is 36,000 OID polls per second, 30x the current rate, before retries and walks. That is not just collector capacity; each leaf's SNMP agent and CPU must answer far more often, which explains the previous spikes. Model-driven telemetry is push-based: the switch samples a sensor path and streams compact updates on a schedule. For counters, a 10-second periodic subscription is appropriate; on-change is wrong because counters change constantly. The data volume is modest if encoded efficiently: a few counters per interface, not 300 individual OIDs. Telemetry also avoids SNMP OID translation and gives structured keys. The change board should see SNMP as the wrong tool for high-frequency counters, not merely a scale-up cost.
2. Choose dial-in or dial-out, transport, encoding, and subscription type. Justify the choice against TLS, load balancing, and the WAN constraints.
Read the worked answer
Choose gNMI dial-out from the switches to the collectors behind the load balancer. Dial-in would require the collector to initiate and maintain sessions to 1,200 switches, open inbound paths, and manage per-device TLS; dial-out lets each switch connect outbound over TLS, which fits security and remote-site operations. Use gRPC/gNMI with GPB encoding for compact, schema-typed data; JSON is acceptable for debugging but costs bandwidth and CPU. Define a subscription with a 10-second sample interval for interface counters and a separate on-change subscription for operational state such as interface up/down. Use sensor paths for /interfaces/interface/state/counters (or the NX-OS equivalent) and select only the needed counters. Configure two collector destinations per switch for redundancy if the platform supports it; the load balancer health-checks collectors. Authenticate with per-switch certificates or AAA-backed credentials and TLS. Test on a canary leaf first; monitor switch CPU, gRPC queue drops, and collector ingest rate.
3. What happens when a collector or WAN path fails, and how should the design handle data loss and backpressure?
Read the worked answer
Dial-out has no large on-switch buffer; if both collectors are unavailable, samples are lost. Design for acceptable loss rather than pretending telemetry is reliable storage. Use two destinations and a load balancer with health checks; if a collector fails, the remaining one absorbs the load and the switch reconnects with backoff. Size collectors for N+1, not exactly the current rate. Monitor subscription state, gRPC errors, and sequence gaps. For 48-hour capacity history, aggregate at the collector and store rollups; do not rely on the switch for replay. Keep a low-frequency SNMP or REST poll as a backup for inventory and health, not for 10-second counters. Avoid adding a third and fourth destination per switch just for redundancy; each destination costs switch CPU and bandwidth. Document that a switch isolated from both collectors during a WAN failure will have a data gap, and mark that gap in reports rather than interpolating.