Replacing 15-Minute SNMP Polling with Model-Driven Telemetry
You operate 2,000 devices and have an SLA requiring interface-down detection within 30 seconds. Today SNMP polling runs every 15 minutes and SNMP traps are unreliable during storms: a recent fiber cut produced 4,000 traps in 90 seconds and the trap receiver dropped about 60% of them. Route-processor CPU reaches 60% during polling windows. You have budget for four collectors and can place them in two data centers. Device mix: 1,200 IOS-XE 17.6, 500 NX-OS 10.2, 300 IOS-XE 16.12. The 16.12 devices support model-driven telemetry but your team has never used it. The NOC wants a single event stream into an existing Kafka topic and wants to keep SNMP as a fallback. Security requires encrypted transport and per-device authentication.
1. Choose between dial-out and dial-in telemetry and justify the choice for this fleet.
Read the worked answer
Dial-out fits better.
- Dial-out: devices initiate sessions to collectors, so you avoid inbound reachability to 2,000 devices, and four collectors behind a load balancer can absorb the fleet. NAT and firewall rules stay simple, and per-device certificates authenticate each publisher.
- Dial-in: collector-initiated gNMI subscriptions give central control and easier subscription changes, but four collectors opening 2,000 sessions is heavy, and every device must be reachable from the collectors.
Given the SLA, security requirements, and collector budget, use dial-out with TLS and per-device certificates, and keep dial-in for troubleshooting or for a small set of devices that cannot dial out. For the 300 IOS-XE 16.12 devices, verify the exact telemetry support and encoding (KV-GPB versus JSON) before committing. If they cannot meet 30 seconds, keep them on a faster SNMP poll (for example, 60 seconds) plus syslog correlation rather than pretending they are covered.
2. Design the subscriptions and the collector-side handling so the NOC gets one reliable stream without recreating the CPU problem.
Read the worked answer
Subscriptions:
- Interface operational state: on-change, event-driven subscription. That gives sub-second detection and avoids constant data. Add a heartbeat sample every 60 seconds so the collector can distinguish no events from dead telemetry.
- Counters and utilization: 30-60 second sample interval, because those are trends rather than alarms.
- Keep paths narrow (interface state plus selected counters) to protect route-processor CPU. Broad subscriptions can recreate the 60% CPU problem you are trying to solve.
- Publish into one Kafka topic with a stable envelope: device ID, path, timestamp, sequence number, collector ID.
Collector handling: deduplicate on (deviceId, path, sequence), handle out-of-order events, and use bounded queues with drop-oldest for state topics. Alert on queue depth and on missing heartbeats. Telemetry over TCP still fails in ways that look like loss when collectors restart, so the pipeline must be idempotent and tolerate replays.
3. How would you cut over from SNMP and prove the new path meets the SLA?
Read the worked answer
Cut over incrementally:
- Pilot 20 devices per platform with telemetry and SNMP running simultaneously; compare detection latency and event volume.
- Use syslog as a cross-check for interface-down, not as the primary signal. During a storm, rate-limit and correlate by device instead of relying on raw trap delivery.
- Keep SNMP at 5 minutes as a fallback for devices without telemetry, and a 60-second poll for the 16.12 group if telemetry is not viable.
- Run failure injection: stop a collector and confirm the load balancer reassigns devices; partition a collector from Kafka; confirm the NOC still sees events within 30 seconds or raises an alert.
- Define a telemetry-stale alarm: no heartbeat for three intervals opens a ticket rather than showing green.
Security validation: per-device certificates, TLS, and authorization so a device can publish only its own paths. Measure end-to-end latency from device event to Kafka consumer, not just collector receipt.