Nightly Config Push Fails on Two Devices After PyEZ Upgrade
A team runs a Junos PyEZ script from a CI runner every night at 02:00 to push a standard set system ntp server stanza to 140 devices. After upgrading PyEZ from 2.5 to 2.7, 138 devices succeed but two MX routers fail with ConnectTimeoutError at the NETCONF handshake. The script uses a shared Device(host=..., user=..., password=..., port=830) factory and a single jnpr.junos.utils.config.Config context per device, sequentially, with no explicit timeouts. The two failing routers are reachable by SSH on port 22 and answer ICMP, but show configuration netconf shows ssh is not enabled under system services. A junior engineer proposes setting timeout=600 and retrying. The team also has a Salt proxy minion setup on the same devices that reports them as Not Connected.
1. Diagnose why increasing the PyEZ timeout will not fix the failure, and state what the two devices actually need.
Read the worked answer
The symptom is ConnectTimeoutError at the NETCONF handshake, not mid-operation. PyEZ connects over NETCONF over SSH on port 830 by default. show configuration netconf shows netconf is not enabled under system services, so the device is not listening on 830 at all. A longer timeout only lengthens the wait before the same failure.
The fix is on the device, not the client: enable set system services netconf ssh (and commit). The fact that SSH on port 22 answers is consistent - that is the CLI, not NETCONF.
This also explains the Salt proxy minions showing Not Connected: the proxy minion's connection to the device uses the same NETCONF transport, so it fails for the same root cause. Do not chase two separate bugs - one change fixes both.
2. The team wants the script to fail fast and clearly when a device lacks NETCONF. What should the script check, and how should the error be surfaced?
Read the worked answer
Before attempting a config push, the script should:
- Attempt
Device.open()inside a try/except that catchesConnectError/ConnectTimeoutErrorand anyRpcErrorfrom the handshake. - On failure, record the device, the exception class, and a hint (
check system services netconf ssh) into a structured result (e.g., a list of dicts or JSON report). - Exit non-zero if any device failed, so the CI job is red.
A short explicit timeout (e.g., timeout=30) is appropriate for the handshake so a missing service fails in seconds rather than minutes. The 600s figure is only sensible for long-running commits, not for connection establishment.
Do not silently continue. The 02:00 job's value is that drift is caught; a green build that skipped two devices is worse than a red one.
3. A reviewer says 'just wrap everything in retry with backoff.' Give a concrete reason that is wrong here and a case where retry is right.
Read the worked answer
Retry is wrong here because the failure is deterministic and configuration-dependent: the device has no NETCONF service. Retrying 138→140 times just multiplies the wait and produces the same error. Retries are for transient failures.
Retry is right for, e.g., a commit that returns a transient lock error (configuration database locked by another user) or a brief NETCONF session drop during a commit on a busy device. In those cases, a bounded retry (2–3 attempts with backoff) is reasonable, but only if the operation is idempotent - re-running a load merge + commit is safe, whereas re-running a commit confirmed without care can leave the device in an unexpected state. Rule of thumb: retry transient errors, fail fast on misconfiguration, and never retry non-idempotent operations blindly.