Polling full configuration every 30 seconds overloads 200 routers

The script is asking for configuration, but the requirement is operational state. A full running configuration is large, includes every policy, route, and interface stanza, and forces the routing engine to serialize and the collector to parse far more data than needed. That explains the CPU load and the growing lag. Fetching the whole config every 30 seconds also creates needless session churn.

Use an operational RPC such as get-interface-information with terse=True and, where supported, a specific interface name. For counters, request statistics for the target interfaces rather than the entire configuration. With NETCONF, apply a subtree filter to the operational data, not to the configuration datastore. With PyEZ, dev.rpc.get_interface_information(terse=True) or dev.rpc.get_interface_information(statistics=True) returns a much smaller document. If using REST, call the equivalent RPC endpoint and request JSON. Regardless of transport, retrieve only the 40 interfaces and the required leaves.

NETCONF is well suited to configuration and structured RPCs, but it is still request-response polling. REST is convenient over HTTP and can return JSON, but it does not inherently reduce device work; the same RPC and filtering discipline is required. Streaming telemetry, such as gNMI, pushes operational state from the device to a collector on a subscription, avoiding a full request every 30 seconds and reducing collector fan-out. It is the best long-term fit for 200 devices and frequent counter updates, provided the platform and Junos release support the required sensors and the team can operate a collector.

For an immediate fix on Junos 20.4R3, start with targeted operational RPC polling, because it requires the least new infrastructure and directly removes the full-config cost. Design the code so the data source is abstracted, then migrate high-frequency counters to streaming telemetry. Do not choose REST merely because it is HTTP; the inefficient query, not the protocol, is the main problem.

from jnpr.junos import Device
from jnpr.junos.exception import RpcError, ConnectError

def get_counters(host, user, password, interfaces):
    try:
        with Device(host=host, user=user, password=password,
                    gather_facts=False, timeout=20) as dev:
            rsp = dev.rpc.get_interface_information(
                terse=True, statistics=True)
            return rsp
    except ConnectError as exc:
        return {'error': 'connect', 'host': host, 'detail': str(exc)}
    except RpcError as exc:
        return {'error': 'rpc', 'host': host, 'detail': str(exc)}

Parse the returned XML by interface name, and treat an interface in the requested list that is absent from the response as missing rather than as zero counters. Distinguish a device that cannot be reached from an interface that is down or not present. Add a per-device timeout and a bounded retry for transient connection failures, but do not retry authorization errors. If one device fails, record it and continue the collection job so that one bad device does not delay all 200.