Automation Rollback Blast Radius Across 1,200 Devices
A managed-services provider operates 1,200 Huawei routers and switches running V200R019, V200R021, and V200R023. A Python/Ansible pipeline uses NETCONF over SSH and YANG models to enforce a baseline: disable Telnet, enable SSH, set NTP, and configure AAA. A compliance run touched all devices in one batch. On 37 devices, the ietf-system and vendor AAA models differed, so the candidate configuration was rejected. The orchestration framework treated any failure as fatal and rolled back the entire batch by restoring a pre-change startup configuration to all 1,200 devices. The rollback reset OSPF adjacencies and caused a 12-minute outage in three regions. The provider has 14 device models and three maintenance windows per week, but no lab covering all versions. The provider wants to resume weekly automation without increasing risk.
1. Identify the design flaws in the automation pipeline and propose a safer execution model.
Read the worked answer
The pipeline treats 1,200 devices as one transaction. It does not discover device capabilities or YANG model versions before rendering configuration, so V200R019 and V200R021 devices receive models they cannot accept. It also has no canary, no per-device candidate commit, and a global rollback that restores startup config and resets OSPF. Safer model: put devices into version and role groups; query NETCONF <hello> capabilities and YANG modules per device; render with conditional templates; run dry-run diff; commit per device or small batch using candidate datastore and confirmed commit; use a 1–5% canary first, then expand only after post-checks pass. Rollback should revert only the changed configuration on the failed device or batch, using a checkpoint or candidate rollback, never a fabric-wide startup restore. Add post-change checks for OSPF neighbors, BGP sessions, SSH reachability, and AAA login before declaring success. Schedule changes in maintenance windows by region to cap blast radius.
2. How should the team handle model differences and rollback so OSPF is not reset?
Read the worked answer
Handle model differences by abstracting intent into a vendor-neutral data model and mapping it to per-version templates, or by using iMaster NCE templates with if/else on software version. For each device, compare the live YANG schema against the template schema; if a node is missing, skip that feature and raise a compliance exception rather than failing the whole batch. For rollback, take a configuration checkpoint before the change and use per-device commit confirmed with a timer; if post-checks fail, roll back only that device. Do not restore startup config on healthy devices, because that is what reset OSPF adjacencies. Test the exact V200R019/V200R021/V200R023 combinations in a lab, and keep a device inventory with model, version, and feature flags. For OSPF, avoid commands that restart the process; use graceful changes and verify display ospf peer after each batch.