Configuration Drift and a Half-Applied Playbook Across 60 Switches
You manage 60 Catalyst 9300 switches with Terraform (network provider) for VLANs and Ansible for interface configuration. Terraform state is in S3 with locking, but two engineers routinely run terraform apply from laptops. Ansible uses ios_config with lines: and no parents:, so re-running the access-switch playbook appends duplicate permit entries to ACLs. A partial run failed on switch 37 and left OSPF neighbors down on 8 switches because the playbook changed MTU before verifying the routing protocol. check_mode reports changes on every run because the device rewrites the running config with different ordering. A manual change added a VLAN that Terraform now wants to delete. Terraform and Ansible both claim ownership of VLAN 40. The team wants one source of truth, but the network is 24x7 and a full re-apply is not allowed during business hours. There is no CI for network changes.
1. Define the ownership boundary between Terraform and Ansible, and explain how to prevent Terraform from deleting the manually added VLAN.
Read the worked answer
Pick one owner per object. Terraform should own device inventory and VLANs (declarative, stateful, plan preview), and Ansible should own interface and ACL policy only if Terraform cannot express it; otherwise move both to one tool. Dual ownership of VLAN 40 is the root cause of the deletion risk: Terraform sees the manual VLAN as drift and plans to remove it.
Import the manual VLAN into state with terraform import, add a lifecycle prevent_destroy for critical VLANs, and require review of any plan that destroys a VLAN. Do not let Terraform manage interfaces with the same provider if it cannot represent the full configuration; partial management causes fights. Document a RACI: who may create a VLAN, who may change an ACL, and how an emergency change is back-ported into code within 24 hours. For the 24x7 constraint, split apply into a small reviewed change set using -target for the specific resource, run during a window, and never apply from a laptop: use CI with state locking and a plan artifact that a second engineer approves.
2. Fix the non-idempotent ACL changes and the unsafe partial failure. Describe the specific Ansible controls and ordering.
Read the worked answer
Make the Ansible task idempotent by specifying parents: so ios_config replaces the ACL block rather than appending permit lines; use match: exact or replace: block where appropriate, and pass the full intended ACL, not a delta. Test idempotence: the second run must report zero changes.
For safe partial failure, use IOS-XE configuration sessions or commit confirm with a rollback timer, so a bad MTU change auto-reverts if the device loses management reachability. Order changes: underlay/routing first, verify OSPF neighbors and reachability, then change MTU or ACLs. Use serial: 1 or small batches, max_fail_percentage: 0, and any_errors_fatal so the play stops before switch 38 when 37 fails. Add pre-checks and post-checks parsed into assertions between stages. For check mode, do not trust the ios_config diff when the device rewrites ordering; compare a normalized parsed running config against the intended config. Keep per-device backups and a tested rollback playbook. A partial run that leaves OSPF down is worse than no run.
3. Design continuous drift detection and a change workflow that avoids business-hours full re-applies.
Read the worked answer
Drift detection should be continuous and read-only: a scheduled job pulls structured config via NETCONF/RESTCONF (or facts plus parsed running config) and compares it to intended state, reporting only semantic differences after normalization. CLI text diffing produces noise because IOS rewrites ordering. Run terraform plan on a schedule with the same state lock, and alert when the plan is non-empty; do not auto-apply.
Change workflow: an engineer opens a PR; CI runs terraform plan and Ansible check mode against a canary switch, posts the diff, and a peer approves. Apply from CI only, with state locking, during a window, to the canary first, then the fleet in batches. Import unmanaged objects rather than letting Terraform delete them. For emergency changes, allow break-glass with a ticket and a 24-hour back-port SLA, then run drift detection to confirm convergence. This gives one source of truth without a big-bang re-apply during business hours.