30-Second Counters on 4,800 Devices Without a Polling Storm

Start from the sample rate and multiply. If the average device has 500 interfaces and you collect four counters each (in-octets, out-octets, in-errors, out-discards) at 30-second intervals, that is 2,000 counters per device every 30 seconds, or about 67 samples per second per device. Across 4,800 devices that is roughly 320,000 samples per second, about 19 million samples per minute. At roughly 150 to 250 bytes per encoded sample on the wire, that is on the order of 60 to 80 MB/s, or 500 to 650 Mbps of sustained telemetry ingest, before the collector does any storage or aggregation. Compare that with today's SNMP: 6,000 OIDs every 300 seconds is about 20 polls per second total, which is why the collectors are CPU-bound on poll processing rather than on data volume.

Bring the number down with four decisions:

1. Do not sample everything at 30 seconds. Restrict the fast interval to customer-facing and uplink interfaces, perhaps 30 percent of ports, which cuts volume by two thirds.
2. Use on-change subscriptions for state that does not need a fixed cadence (interface oper status, BGP peer state, optics alarms) instead of polling it.
3. Push aggregation to the collector tier: store 30-second raw data for 24 to 48 hours and roll up to 5-minute for the long retention the NOC actually uses for capacity planning.
4. Use dial-out from the device to a collector fleet with horizontal sharding, so 4,800 devices are not opening sessions to one endpoint.

Only after that arithmetic is on paper should you pick the encoding and transport.

The rollback problem in the scenario is not a scripting problem; it is a state problem. A rollback script fails when the device is not in the state the script assumed, so capture the actual pre-change state and make the device responsible for reverting.

For model-driven devices (IOS-XR and NX-OS):

1. Keep intent in a source of truth, expressed per device role, not per device CLI text.
2. Render candidate configuration and diff it against the running datastore before anything is committed. The diff is the change record and the impact statement.
3. Use the candidate datastore with commit confirmed and a short confirmation timer. If the change breaks management reachability or the post-check fails, the device reverts itself without the automation being in the loop.
4. After commit, run a post-change validation (interfaces up, BGP sessions established, expected routes present). On failure, issue the explicit revert and record the outcome.

For the 700 CLI-only devices, the honest answer is that you cannot get transactional rollback, so compensate:

• Take a full configuration backup immediately before the change and store it with the change record.
• Use an idempotent tool (Ansible or NAPALM) with structured parsing rather than expect scripts, so the pre-check reads actual state and aborts on drift.
• Apply the smallest possible change set, and verify each device before moving to the next; never batch 700 devices in one run.
• Put these devices on a replacement roadmap, because the compliance report will expose how much of the fleet cannot be verified.

Gate everything through a CI check that runs the diff and validation against a lab instance, and a canary group of 20 devices with a soak period before fleet rollout.

Trust is rebuilt by making the monitoring disagree with reality less often, and by making gaps visible rather than silent.

1. Instrument the pipeline itself. Publish a data-freshness metric per device and per collector, and alarm when a device has not reported within three intervals. Today 11 percent of polls vanish and nobody knows which ones, which is exactly why engineers log in. A visible "no data since 09:14" is far better than a graph that quietly flatlines.
2. Move state away from polling. Interface oper status, BGP/IS-IS adjacency state, and optics alarms should arrive as on-change telemetry or traps, so a link-down event is reported in seconds instead of at the next 5-minute poll.
3. Use 30-second counters where they change decisions: customer SLA reporting, congestion detection, and microburst evidence. Do not apply them to every port; wasted resolution costs collector capacity and adds nothing.
4. Close the loop with the configuration pipeline so the compliance report is derived from the same source of truth that generates configuration. A daily report that compares intended versus running state, with a diff per device and an owner per exception, turns compliance from a spreadsheet into an actionable queue.
5. Retire the manual check by giving the NOC a single view that combines streaming counters with the current adjacency and configuration state; if the view is right, the CLI habit fades.

Sequence the build: collectors and freshness monitoring first, then on-change state subscriptions, then 30-second counters on the agreed subset, then the daily compliance report, then automated remediation for the drift classes with the lowest risk.