The Monthly Standardization Playbook That Broke Twelve Leaf Pairs

replace: block is a overwrite, not a merge.

It takes everything between the begin and end markers and substitutes the rendered template output. Any line that exists inside those markers but is absent from the template is deleted. The template contains descriptions, switchport mode, access VLAN, and MTU - it does not contain vpc 10, channel-group, or spanning-tree port type edge trunk. On switches where the hand-written markers happened to span interfaces that are vPC member ports, the render silently deleted the vPC configuration from those ports.

The second run amplified the damage because of partial application. With default settings there is no serial and no max_fail_percentage, so the run had already completed on sixteen devices when it timed out on seventeen. Re-running from the top applied the template to a different subset again. Net effect: within a vPC pair, one peer has the new MTU and STP settings and the other does not.

That divergence is exactly what produces peer-link flaps - vPC type-1 consistency checks MTU and STP port type, and a mismatch puts the peer-link or member ports into a suspended state. The flapping is not a side effect; it is the predictable output of half-applying a global parameter across a pair.

Ranked by damage prevented:

1. A diff review before write. --check --diff on a single canary switch would have printed every - vpc 10 line as a deletion. This alone stops the entire incident, because the operator sees the deletion before it happens anywhere. Highest value by a wide margin.
2. Configuration backup as part of the task (backup: yes or an out-of-band pre-change capture). This does not prevent the incident but converts recovery from reconstruction to restore, and it makes the blast radius bounded.
3. Blast-radius control: serial with a small batch size and max_fail_percentage: 0. This turns a timeout on switch 17 into a hard stop rather than a half-applied fabric, and prevents the re-run from re-shuffling which devices are configured.
4. Scoping the template to intent, not to a block. Excluding vPC member ports from the managed range, or using resource modules that manage declared properties rather than replacing a text block, removes the class of error entirely.
5. Idempotency verification: a second run reporting zero changes on a device that was already compliant would have flagged the divergence in early months, before twelve pairs were affected.

Note the ordering principle: the controls that put a human between the render and the device rank above the controls that merely make recovery faster.

Do not re-run the playbook. Freeze it first.

1. Freeze automation. Disable the scheduled job and any pipelines that call this playbook. A scheduled run landing mid-recovery will re-apply the faulty template to devices you have just fixed.
2. Inventory divergence per pair, not per device. On each of the four pairs run the vPC consistency check to see which parameters disagree - MTU and STP port type are named in the symptoms, but check VLANs and port-channel settings too. A pair is only recovered when the check is clean, not when the config looks similar.
3. Recover the secondary side first, leaving the primary untouched where possible. Bring the pair into consistency with the vPC primary in a controlled window, since restoring the peer-link brings suspended member ports back and can cause a broadcast/STP reconvergence.
4. Restore from backup where reconstruction is ambiguous. If the deleted vpc and channel-group lines cannot be reproduced exactly from another pair's config, restore the pre-change config instead of hand-editing; hand-editing under pressure is how the second divergence gets introduced.
5. Fix the template and the markers so the managed range cannot include vPC member ports, and delete the hand-written markers in favour of a machine-generated or resource-module approach.
6. Resume with a canary. Run --check --diff against two switches in different vPC pairs, read every deletion line, then apply to the canary pair with serial: 1 and max_fail_percentage: 0, verify vPC consistency, and only then widen.

The resumption gate is not 'the playbook ran green' - it is 'the diff showed no deletions of vPC, channel-group, or STP configuration on a canary pair, and the consistency check is clean after apply.'