Reconciling 61 drifted nodes in NSP after an incident workaround

The regional team is right, and the automation team's proposal misreads what the drift contains.

The 340 changes are not random drift. Most are deliberate, currently load-bearing workarounds around an active fiber cut: raised IS-IS metrics on the failed path, shortened BFD timers, and static routes holding traffic on the surviving path. Reconciling to intent would restore the pre-incident metrics and timers and remove the static routes while the fiber is still down, which risks pushing traffic back onto a degraded or dead path, causing a second reconvergence across 61 nodes, and potentially blackholing the services the workaround is protecting. A single job applying a full configuration diff to 61 nodes at once also means 61 simultaneous opportunities for a session reset, with no canary and no per-node checkpoint.

The real risk therefore is not that reconcile produces incorrect syntax; it is that the system cannot distinguish approved exception from drift. Triage before touching anything: export the per-node diff from NSP, classify each change as workaround, cosmetic or genuinely unknown, and get the regional team to annotate the workaround set with an expiry tied to fiber restoration. Reconcile only the categories that are provably safe as a first pass; leave the workaround set alone until the incident is closed.

Sequence matters more than the tooling.

1. Freeze the intent templates for the affected services and record the current running configuration of all 61 nodes as a checkpoint, so any step is reversible.
2. Run the reconcile in report-only or advisory mode first and diff its output against the triaged change list from the regional team. Anything the job would revert that is on the approved-exception list is excluded before the job ever writes.
3. Encode the exceptions as a template or policy overlay with an expiry date, so that the exception is now the intent rather than a permanent blind spot.
4. Apply to a canary of two non-critical nodes, with pre- and post-checks on IGP neighbor counts, BFD session state, BGP session state and service-level pings. Only then widen the blast radius in small batches, with a pause between batches for the checks to run.
5. Do not run any of this in the 72 hours before the NSP upgrade. The upgrade changes mediation and model handling, and any reconciliation state you create now may be re-evaluated by different software. Upgrade first, verify synchronization across all 420 nodes, then reconcile.
6. Close the loop: drift detection should route to an owner with a review queue, not to an automatic remediation job.

Partial model support means NSP cannot reliably render or validate the full configuration for those nodes, so they must not be treated as fully managed devices. If a template touches a container that the supported model does not cover, the push can either fail or, worse, succeed while silently dropping the uncovered configuration.

Practical handling:

• Move the five nodes into a separate managed group with an explicit label, so that templates are scoped and cannot accidentally apply to them.
• Fall back to a mediation path that is actually supported for that release: SNMP-based polling for monitoring with CLI-based, reviewed configuration changes, accepting that drift detection will be coarser and that some configuration will be invisible.
• Restrict those nodes to the subset of templates whose objects are fully modelled, and document the excluded objects so nobody assumes coverage exists.
• Treat the software upgrade of those five nodes as the real remediation, and put it on the plan with a target date; partial model support is a release mismatch, not a permanent architecture decision.
• Until then, exclude them from any automated reconcile job, because an unreconcilable node that the job thinks it can reconcile will generate perpetual false drift and erode trust in the whole system.