Workflow fails on three of twelve nodes during a VPRN rollout

Diagnosis: SR OS 23.10 defaults to MD-CLI while the template emits classic CLI. The configure service vprn 100 ... tree is syntactically different in MD-CLI, so the node rejects step 4. The nine-pass/three-fail split correlates exactly with the software version, which is what rules out CSV data errors or per-node reachability problems. NSP, the CSV and the transport are all fine; the dialect is the fault.

Immediate recovery:

• Stop the queue before the remaining five nodes run, and confirm their versions from NSP inventory rather than assuming.
• On each failed node, check show service id 100 and remove the partial VPRN after confirming no SAP or BGP configuration landed. A half-built VPRN with an RD and no interfaces is a latent trap for whoever troubleshoots next.
• Re-run the three nodes with an MD-CLI variant of the template, or use NSP's model-driven service creation so the platform handles the dialect.
• Record the deviation in the change record: the run is no longer a repeatable standard until the template is corrected, and the audit requirement is already affected.

• Model-driven first: use NETCONF/YANG-based intent or NFM-P service templates instead of raw CLI text; the platform adapts to the node's management interface. If CLI templates must stay, split them by major release and select by a version attribute pulled from NSP inventory.
• Idempotence: every step reads state before writing, creating or modifying only what is missing, so a retry after a partial failure converges instead of erroring or duplicating.
• Pre-checks: node reachable, software version in the supported set, free SAP/port capacity, RD and RT uniqueness, BGP peer defined. Fail fast on data errors before any config is sent.
• Rollback: capture the pre-change configuration per node and define an explicit rollback step; abort-on-error alone guarantees partial state, as this incident showed.
• Canary and thresholds: run one node per version, validate, then fan out; stop after two failures rather than at the first, so the operator learns the scope quickly.
• Audit trail: log each step's request and response to the workflow log so the change is reproducible and reviewable, and tie it to the change record.

'Workflow completed' is not verification. Per node, check:

• VPRN 100 exists and is administratively up, with both SAPs operational.
• The BGP session to the route reflector is established and the expected remote prefixes are present in the VPRN route table.
• A synthetic ping or service OAM test from a CE-facing SAP succeeds end to end.
• The running configuration matches the intent/golden template - NSP configuration audit or a CLI diff, not eyeballing.
• No residual partial configuration remains on the three nodes from the first attempt.

Aggregate the results as pass/fail per node per check so there is a single evidence pack for the audit. Then close the loop: enable assurance so later drift from the intent is alarmed rather than discovered at the next change window, and add the version attribute to the compliance check so a future software upgrade flags affected templates before it is applied in production.