Nightly NETCONF Drift Job on a Mixed IOS XE Fleet
Your team runs a nightly Python compliance job against 520 IOS XE routers. It opens one NETCONF session per device, calls on the running datastore filtered to native/router/interface, then sends an that rewrites every interface description and shutdown state. After a month, 31 devices return lock-denied, 12 return operation-failed after partial writes, and the job has twice left interfaces administratively down. Device CPU reaches 80 percent during the run. The fleet is mixed: 480 devices run 17.6 with the candidate datastore and confirmed-commit enabled; 40 run 16.9 where policy disables the candidate datastore. The job also retrieves the entire interface tree, including counters, and compares XML strings. The job has no backoff and retries immediately twice. Your manager wants drift corrected faster without another outage and asks whether moving to RESTCONF will fix it.
1. Diagnose the lock-denied and partial-write failures, and explain what should change in session handling, datastore reads, and edit strategy.
Read the worked answer
Lock-denied is a contention and lifecycle symptom, not a permissions problem. A prior session may still hold the running lock after a dropped TCP connection, or two scheduler instances overlap. Immediate retries amplify the contention: both retries race the stale lock and each other. Fix by using a bounded lock timeout, jittered exponential backoff, and a single-flight lock per device. Always release the lock in a finally block and close the NETCONF session cleanly.
Partial writes come from batching unrelated interface edits into one edit-config. Many platforms process edits sequentially; an error halfway leaves earlier changes applied. Use error-option='rollback-on-error' where supported, or split the edit into per-interface transactions with a lock held across the set. Prefer minimal diffs: send only changed nodes with operation='merge', and explicitly delete nodes that must disappear. Do not rewrite every description and shutdown state.
Reads are too heavy. <get-config> on running is correct for config-only data, but the filter must exclude counters and state. Retrieve only the interface subtrees under management, and compare normalized structured data rather than XML strings. String comparison creates false drift from namespace prefixes, whitespace, and element order, which then triggers unnecessary edits and load.
2. Design an idempotent edit workflow that works on both the 480 candidate-capable devices and the 40 devices where the candidate datastore is disabled. Include confirmed-commit use and rollback.
Read the worked answer
For the 480 devices with candidate datastore: lock the candidate datastore, apply only the computed diff, run <validate>, then <commit> with a confirmed-commit timeout of 60 to 120 seconds. Verify reachability and the intended interface state, then send <commit> again to confirm. If verification fails, send <discard-changes> or let the timer expire. This gives atomic apply and automatic rollback without touching running until validation succeeds.
For the 40 devices without candidate datastore: lock running, take a pre-change config backup, and apply the minimal diff with rollback-on-error if the platform supports it. Where rollback-on-error is unavailable, apply changes in small per-interface transactions and verify after each. You cannot safely use confirmed-commit on these devices; use a configuration archive or configure replace with a revert timer if the platform supports it, and keep an out-of-band path.
Idempotency rules: build the edit from desired state versus current state, include only changed leafs, use explicit delete operations for removed nodes, and make shutdown state explicit. Render the same desired state twice; the second pass must produce an empty diff. Canary five devices per platform and software version before the fleet run. This design removes the partial-write class and makes retries safe.
3. Will moving from NETCONF to RESTCONF fix the outage and contention? Justify the decision and state any conditions.
Read the worked answer
No. RESTCONF is an HTTP mapping to the same YANG datastores and has the same lock, candidate, and commit semantics. Moving the transport does not make a multi-interface edit atomic, does not remove stale locks, and does not create idempotency. A poorly designed RESTCONF job can still overwrite unchanged leafs and leave partial state.
RESTCONF can be a reasonable choice when you need HTTP-friendly tooling, JSON payloads, or when NETCONF is blocked by policy. It may reduce XML parsing complexity and fit existing API gateways. But it also adds HTTP retry and timeout behavior that can duplicate non-idempotent operations, and not every IOS XE image enables the RESTCONF server. Confirm image support, AAA, and TLS first.
Keep NETCONF for transactional configuration on this fleet. Use RESTCONF selectively for read-only inventory or where the device exposes the needed model only over RESTCONF. The real fixes are minimal diffs, locking discipline, backoff, candidate datastore where available, confirmed-commit on capable devices, and canary verification. Those changes address the symptoms regardless of transport.