Weekly CAB versus automated change with segregation of duties
A 30-person network engineering group supports 400 branch offices. Changes currently follow a weekly change advisory board: engineers submit a ticket, wait up to seven days, then apply CLI changes during a 4-hour window. The group averages 60 changes per week and has had two Sev-1 outages this quarter, both traced to manual copy/paste errors in firewall filter terms. Internal audit requires segregation of duties between the person who writes a change and the person who approves it, plus an auditable record of what was deployed. The director wants a blameless postmortem culture, but one engineer was recently blamed in a team meeting for an outage. Four engineers are certified on Junos automation and can build a pipeline, but the other 26 are CLI-only and skeptical. The company uses GitLab, Ansible, and has a small lab of four vSRX devices.
1. Design a change workflow that meets segregation of duties and shortens lead time. Explain which controls are automated versus human, and why.
Read the worked answer
Separate the roles rather than the calendar. The author creates a merge request; a different engineer approves it under a protected-branch rule. The deployment runs as a service account from the merged commit, not from the author's workstation, so the person who wrote the change cannot also execute it without review. GitLab records the diff, approvals, pipeline logs, and deployer identity. The device commit comment should include the merge request ID and commit SHA, giving an auditable link from Git to the device.
Automate low-risk standard changes: syntax checks, policy checks, rendering, commit check, and deployment. Reserve human approval for the merge request and for a small high-risk category such as management-plane filters or routing policy that can drop the control plane. This keeps separation of duties because the approver is not the author, while avoiding a weekly queue for routine interface and filter changes. The CAB becomes a review of exceptions and failed changes, not a gate for every change.
2. How should the postmortem for the firewall filter outage be run to support learning while satisfying audit? Include artifacts.
Read the worked answer
Run a blameless but evidence-based review. State up front that the purpose is to find control and system weaknesses, not to assign personal fault. Build a timeline from the merge request, pipeline log, device commit history, configuration diff, monitoring alerts, and the rollback action. Ask why the filter term was easy to copy incorrectly, why tests did not catch it, and why deployment proceeded without a reachability check. Capture contributing factors rather than a single root cause.
For audit, keep the artifacts and the action items: the MR diff, reviewer identity, pipeline output, show system commit output, the before/after configuration diff, and the time to restore. The audit record can show that segregation of duties existed and that a control gap was found and fixed. Distinguish learning from accountability: no names in the postmortem narrative, but named owners and due dates for actions. Share the action tracker with audit so follow-through is visible.
3. What adoption approach would you use for the 26 CLI-only engineers, and how would you measure whether it is working?
Read the worked answer
Start with read-only automation that makes their current job easier: scheduled compliance reports, configuration backups, and drift summaries. Pair each skeptical engineer with one of the four automation engineers on a real change, so the first experience is solving their problem rather than attending a training class. Provide reviewed templates and a self-service merge request path for common changes. Keep a CLI escape hatch for emergencies but require it to be reconciled back into Git afterward.
Measure adoption by the percentage of changes deployed through the pipeline, the number of active authors outside the original four, median lead time, and change failure rate. Do not use ticket counts alone; a team can game those. Track how often manual changes are later overwritten by automation, because that indicates unresolved drift. Review progress monthly with the director and adjust the high-risk list based on actual incidents.