CI pipeline reports green after removing a management filter
A CI/CD pipeline built in GitLab deploys Junos configuration to 12 EX switches. The pipeline has three stages: lint, deploy, and notify. The deploy job runs an Ansible playbook against all 12 devices in parallel with throttle: 0. A recent merge request removed a management firewall filter from a template. The lint stage only ran YAML lint; the deploy stage succeeded on all devices and reported green. The next morning, 12 switches were unreachable in-band, and the team drove to site. Secrets are currently stored in CI variables, visible to all maintainers, and one former contractor still has maintainer access. The team wants faster feedback but cannot add more than two minutes to the pipeline. The switches are managed in-band only; no console server is in place.
1. Identify the pipeline design failures and propose stage gates that fit the two-minute budget.
Read the worked answer
The pipeline validated syntax but not the rendered configuration, did not run commit check, deployed to all devices simultaneously, and had no post-deployment reachability check. The green status reflected Ansible exit codes, not whether management access survived. throttle: 0 removed the ability to stop after a bad canary.
A workable two-minute design: a fast render-and-lint stage using YAML lint, ansible-lint, and a template render that fails on undefined variables; a policy check that rejects deletion of the management filter unless an explicit reviewed exception is present; a canary deploy to one switch using commit confirmed; an automated reachability check over the management path; then the remaining devices in small batches with a post-check. Parallelize lint and policy checks so they consume well under a minute. Keep the deploy stage from starting until the canary check passes. For management-plane changes, require a second reviewer and an explicit exception label. The pipeline should fail closed, not notify and continue.
2. How would you make the management-plane change reversible and safe, given there is no console server?
Read the worked answer
Use commit confirmed with a short timer, for example 5 minutes, on the canary and on each batch. If the change severs in-band management, the device automatically rolls back when the timer expires. Verify reachability from an independent probe before issuing the confirming commit. Never remove the existing management filter before adding and testing its replacement; stage the new term first, confirm access, then remove the old term in a separate change. Without a console server, the rollback window is the main safety net, so do not use a long timer that exceeds the team's response time and do not run all devices at once. Keep an out-of-band recovery plan: a local technician or a temporary cellular path. Document the exact rollback command and the expected timer behavior in the runbook.
3. What would you do about the exposed CI variables and the former contractor's access?
Read the worked answer
Treat the current credentials as compromised and rotate them. Move secrets to a dedicated secret manager or protected, masked CI variables scoped to the deploy environment, and restrict who can trigger or view deployments. Remove the former contractor's maintainer access immediately, review all maintainers, and enable protected branches so changes need approval by someone other than the author. Use a short-lived deployment identity rather than a shared static password. Add an audit review of recent pipeline runs and repository access. These controls are part of the delivery design, not an afterthought: a pipeline that can push management-plane changes must limit who can change the pipeline and who can read its secrets.