Model-Driven Telemetry and Closed-Loop Assurance at Scale
A service provider manages 3,200 devices from three vendors: 1,800 IOS XR, 1,000 IOS XE, and 400 third-party. The current SNMP poller samples 60,000 interfaces every 5 minutes and cannot support the 10-second interface statistics and 500,000 flow records that the assurance team requires. The collector is a single 1 Gbps server, and the team wants dial-out telemetry rather than device-initiated polling. Configuration is still done with CLI scripts and SSH, causing drift. The provider wants a model-driven automation and assurance architecture that can be introduced incrementally, with a 12-month target. Security requires TLS and per-device authentication, and the network operations center needs a single inventory source of truth.
1. Recommend the telemetry transport and data models for the 10-second interface statistics, and estimate whether the 1 Gbps collector is sufficient. Show assumptions.
Read the worked answer
Use gNMI dial-out with OpenConfig interface and system models for streaming telemetry, and keep NETCONF/YANG for configuration. gNMI is designed for high-frequency streaming and supports dial-out over gRPC with TLS and per-device credentials. NETCONF is better suited to transactional configuration than to 10-second counters.
Assume 60,000 interfaces send a 400-byte update every 10 seconds: 60,000 x 400 = 24 MB per interval, or 2.4 MB/s, about 19 Mbps. If updates average 1 KB, the rate rises to about 48 Mbps. For 500,000 flow records every 10 seconds at 250 bytes each, the rate is 125 MB per interval, or 12.5 MB/s, about 100 Mbps. Combined, the load is roughly 120 to 150 Mbps, so a 1 Gbps collector has capacity, but a single server is not highly available. Deploy at least two collectors behind a load balancer, size CPU and disk for the time-series database, and use sampling or on-change for flows. Validate device CPU before enabling all interfaces.
2. How would you separate configuration and assurance workflows, and what role should NETCONF/YANG play versus gNMI?
Read the worked answer
Separate the systems by intent. Maintain a single source of truth for inventory, device roles, and intended configuration. The configuration workflow uses NETCONF with YANG models, candidate datastores, confirmed commit, and rollback. Templates and CI/CD pipelines generate intended configuration, and a controller reconciles drift rather than allowing engineers to SSH into devices. The assurance workflow uses gNMI dial-out telemetry into collectors and a time-series database, with analytics and alerting reading from that data.
Do not use the telemetry collector as a configuration tool, and do not use NETCONF polling as the primary telemetry mechanism. ZTP can bootstrap devices and establish trust with certificates. For third-party devices, use the strongest available model-driven interface and document gaps rather than falling back to CLI silently. The inventory source of truth should feed both workflows so a device is never configured under one identity and monitored under another. This separation also limits the blast radius of an automation error.
3. Design a closed-loop assurance process that avoids alarm storms and safely remediates a failing interface.
Read the worked answer
Ingest gNMI telemetry into collectors, normalize it, and store it in a time-series database. Apply baselines and correlation before raising alarms: suppress downstream alarms when a parent interface is down, deduplicate repeated interface flaps, and require a minimum duration or error-rate threshold. Enrich alerts with topology and service impact so the NOC sees one incident, not hundreds of interface events.
For remediation, start with read-only detection and a recommendation. When confidence is high, the assurance system can open a change request to the automation controller, which validates intent against the source of truth and uses NETCONF confirmed commit with a rollback timer. After the change, verify recovery from telemetry before closing the loop. Keep guardrails: no remediation on core links without approval, rate-limit actions per device, and log every step. For a failing interface, the safe sequence is to check optics and errors, move traffic if a protected path exists, and only then reset or replace the interface. The closed loop must be able to stop and roll back automatically if telemetry does not recover.