A CI/CD Pipeline That Broke 12 Branch Routers
A GitLab CI pipeline runs an Ansible play against 60 branch routers with serial: 1. The deploy job failed at host 13 because a Jinja variable was undefined for one hardware model. The pipeline reported failure, but hosts 1-12 had already received the new ACL; the matching route-map task never ran because the play aborted. On-call restored service manually. The repository also contains a vault password file committed six months ago, and the only tests run against a three-router lab that does not include the failing model. There is no rollback stage; recovery means re-running the play after fixing the variable. The team deploys weekly and wants to move to daily. Change approval is a manual ticket. Secrets are stored as masked CI variables for production, but the lab uses a shared password in group variables.
1. Triage the immediate partial-configuration problem and describe the pipeline changes that would have prevented it.
Read the worked answer
Immediate: identify hosts 1-12, confirm whether the ACL is present and whether the route-map references it, then either roll forward with a corrected variable or restore the pre-change config from a backup. Do not re-run the whole play blind. Ansible will skip tasks that are already satisfied only if they are idempotent, and a partially applied ACL may still be modified.
Prevention:
- Add a pre-flight render stage that evaluates every template for every model in the inventory, including the model that failed. Undefined variables fail there, before any device is touched.
- Separate plan from apply: render intended config, diff against running config, and require an explicit apply job for that diff.
- Add per-device post-change assertions (ACL exists, route-map references it, BGP neighbor remains up) using
block/rescue/alwaysso verification runs even when an earlier task fails. - Record per-host status in an artifact so a partial rollout is visible, not just a red pipeline.
The root cause is not serial: 1; it is that generation, application, and verification were one uninterruptible play with no inventory-wide validation.
2. Redesign the CI/CD stages and test strategy to support a daily deployment cadence.
Read the worked answer
Proposed stages:
- Lint and secret scan: yamllint, ansible-lint, and a secret scanner on every commit.
- Render for 100% of inventory models: unit-test parsed outputs with fixtures, including one fixture per hardware model.
- Lab integration: a topology that includes the model that broke, using containerlab or a virtual device rather than an approximate stand-in.
- Dry-run diff against production: read-only collection, publish the diff for approval.
- Canary: one device, manual approval, post-checks.
- Batch rollout:
serial: 5,max_fail_percentage: 0, pause between batches. - Post-verification and automatic rollback when an assertion fails.
The critical changes are separating plan from apply and making the canary gate mandatory. Daily deploys are safe only after model coverage and canary checks exist; otherwise you are merely increasing the failure rate. Keep the change ticket, but bind approval to the diff artifact hash so the approver approves the actual change.
3. How would you fix the secrets exposure and the approval process without slowing delivery to a crawl?
Read the worked answer
Secrets: remove the committed vault password file, rotate everything it protected, and purge it from history. Move to a secrets manager (HashiCorp Vault, AWS Secrets Manager, or similar) with short-lived credentials, and use GitLab OIDC to obtain them rather than long-lived masked variables. Separate lab and production credentials entirely; the shared group password is a blast-radius problem. Scope production credentials to the deploy job and protect the environment so only authorized jobs can obtain them.
Approval: keep the ticket but bind it to the diff artifact hash. For daily cadence, auto-approve low-risk changes (NTP, banners, descriptions) and require human approval for ACL, routing, and management-plane changes. That policy is a judgement call; write it down, encode it in the pipeline, and test it. A weekly cadence with no canary is not safer than a daily cadence with one; the controls matter more than the frequency.