NDFC vPC Drift and a 90-Minute Upgrade Window
An NDFC instance manages two VXLAN EVPN fabrics: Fabric-A has 4 spines and 40 leaf switches, Fabric-B has 2 spines and 26 leafs. During a recent incident, an engineer manually changed the vPC domain ID on leaf-04 only; leaf-03 still has the old domain. NDFC shows both leafs as out-of-sync under Configuration Compliance, but the fabric is passing traffic and the engineer wants to click Deploy on both switches to clear the alert. The next maintenance window is 90 minutes and must also start an image upgrade from Cisco NX-OS 10.2(5) to 10.3(4a) on all 72 switches, including the vPC pairs. NDFC image management shows the gold image as 10.3(4a), but two leafs are on 10.2(3) and one spine is on 10.4(1). The team has no console access to remote sites and relies on in-band management.
1. Assess the risk of the vPC domain ID mismatch and describe the correct remediation. Explain why clicking Deploy on both leafs immediately is unsafe.
Read the worked answer
A vPC domain ID mismatch is not a cosmetic compliance issue. The vPC pair must use the same domain ID, peer-link, and consistency parameters; otherwise vPCs may be suspended or traffic may silently use one member. Clicking Deploy on both without checking the intended value can overwrite the good leaf and bounce the peer-link during production traffic. First, compare show run vpc, show vpc, and show vpc consistency-parameters on both leafs; determine whether the intended NDFC value is the old or new domain. If NDFC intends the old ID, revert leaf-04 to the old ID during the window. If NDFC intends the new ID, change leaf-03 to match, but do so with the pair in maintenance mode and after verifying the peer-link. Use NDFC Config Preview to see the exact diff; deploy to the vPC pair together, not one switch at a time, and watch vPC status and traffic. Do not combine this remediation with the image upgrade on the same pair in the same step.
2. Plan the image upgrade across 72 switches in the 90-minute window. Address the mixed versions, vPC ordering, ISSU, in-band management, and rollback.
Read the worked answer
First validate the upgrade path. A spine on 10.4(1) cannot be treated like a 10.2(5) device; moving it to 10.3(4a) is a downgrade and is likely disruptive, not ISSU. The two leafs on 10.2(3) may need an intermediate hop. With 72 switches and 90 minutes, a sequential ISSU at ~2-3 minutes per switch is already tight; adding a downgrade makes the window unrealistic. Options: change the gold image to a version supported by all devices, extend the window, or phase the upgrade. For vPC pairs, upgrade one member at a time with ISSU where supported, wait for the peer to re-form and vPCs to come up, then upgrade the other. Upgrade spines and leafs in an order that preserves EVPN peering and in-band management; never upgrade both vPC peers or all spines simultaneously. Use NDFC image management pre-checks, compliance, and rollback image. Keep a console/out-of-band path or staged in-band management plan because a failed upgrade can isolate a remote site.
3. How would you use NDFC APIs and compliance data to prove pre- and post-change state and prevent clearing an alert without real remediation?
Read the worked answer
Automate evidence collection through the NDFC REST API before and after the window. Capture configuration compliance for both fabrics, the running image per switch, and the vPC consistency output; store the JSON as a pipeline artifact. Define a gate: no switch may be out-of-sync, no vPC pair may have mismatched domain/peer-link parameters, and no device may be outside the approved image list. After remediation, rerun the same queries and compare. Do not let the API report alone approve the change; correlate with show vpc and EVPN neighbor state from NX-OS. If NDFC marks a device in progress, wait for the job to complete rather than issuing a second deploy. Use a dedicated NDFC service account with least privilege, and log every deploy action with the change ticket ID. This gives an auditable pre/post record and prevents clearing compliance by deploying without verification.