Intermittent PyEZ session failures against 60 MX routers
Your team runs a Python/PyEZ inventory job against 60 Juniper MX routers every 15 minutes from a single Linux jump host. The script opens a Device session per router, gathers facts, retrieves interface counters, and closes the session. After a month, about 4 percent of runs raise RpcTimeoutError or ConnectTimeoutError; the failures cluster around the same 6 devices, and the jump host shows roughly 900 sockets in TIME_WAIT. A junior engineer proposes wrapping every call in a bare try/except and sleeping 60 seconds on failure. The job must finish within 10 minutes, and operators rely on the counters for capacity alerts. Device login banners, SSH rate limiting, and max-sessions are at default. The script currently uses Device(host=..., user=..., password=...) and no context manager.
1. What is the most likely cause of the clustered failures, and why is a bare try/except with a 60-second sleep a poor fix? Propose concrete session-management changes.
Read the worked answer
The TIME_WAIT count and the same 6 devices point to sessions that are opened and never deterministically closed when an exception occurs. PyEZ opens a NETCONF-over-SSH session; without a context manager or finally: dev.close(), a timeout leaves the socket until TCP times out. The 6 devices likely have lower session limits or SSH rate limits, so retries accumulate. A bare except catches authentication failures, commit errors, and timeouts alike. Sleeping 60 seconds and retrying can trip Junos login retry limits and lock the account, and it hides permanent errors. Use with Device(...) as dev: or a try/finally block, set conn_open_timeout and timeout, disable fact gathering when not needed (gather_facts=False), and close explicitly. Retry only transient exceptions with bounded exponential backoff, for example 3 attempts at 2, 4, and 8 seconds. Reduce thread count and stagger start times so devices are not hit simultaneously. Log device name, RPC, duration, and exception type.
2. Show the core Python structure you would use for one device and for concurrent execution, including exception handling.
Read the worked answer
Use one Device per worker thread; PyEZ Device objects are not safe to share across threads. Keep the session scope tight and always close it.
from concurrent.futures import ThreadPoolExecutor, as_completed
from jnpr.junos import Device
from jnpr.junos.exception import ConnectError, RpcTimeoutError
TRANSIENT = (ConnectError, RpcTimeoutError)
def collect(host, user, password):
for attempt in range(3):
try:
with Device(host=host, user=user, password=password,
gather_facts=False, timeout=30,
conn_open_timeout=10) as dev:
rsp = dev.rpc.get_interface_information(terse=True)
return host, rsp
except TRANSIENT as exc:
if attempt == 2:
return host, f'FAILED: {type(exc).__name__}'
time.sleep(2 ** (attempt + 1))
except Exception as exc:
return host, f'PERMANENT: {type(exc).__name__}'
with ThreadPoolExecutor(max_workers=8) as pool:
futures = [pool.submit(collect, h, u, p) for h in hosts]
for fut in as_completed(futures):
record(fut.result())
The with statement closes the session on both success and failure. The retry list is deliberately narrow so an authentication or authorization error does not get retried.
3. How would you verify the change reduced failures without waiting a month? Name metrics and a test plan.
Read the worked answer
Run a canary against the 6 problem devices first, then a controlled subset, before the full inventory. Compare failure rate, p95 job duration, and open socket count before and after. Add a synthetic check that deliberately triggers a timeout on a lab device and confirms the session is closed and the retry path is bounded. Track TIME_WAIT every minute during the run; it should fall after the job completes instead of climbing. Alert if any device returns a permanent error, if a single device is retried more than three times, or if the job approaches the 10-minute budget. Run the canary for several cycles at the same 15-minute cadence, because the original failures were intermittent and clustered. Also verify account lockout counters on the 6 devices after an induced failure, since the old retry design could have locked accounts.