YAML, Jinja2, and a Green CI Pipeline That Broke 12 Branches
A network team manages 300 branch devices with a Git repository, YAML inventory, and Jinja2 templates. A CI job renders configurations and pushes them with a Python script. A recent merge changed ntp_servers from a list to a mapping and added flow_control: on to the YAML. The rendered config for 45 devices contained ntp server True and flowcontrol on on platforms that require flowcontrol receive on; 12 devices lost management connectivity after the push. CI was green because its only test checked that the output was non-empty and contained hostname. The team uses YAML 1.1 parsers in some tools and JSON in others. The next release must add 80 devices and a new vendor model, and the manager wants a validation gate that catches this class of error before any device is touched.
1. Root-cause the rendered `True` and the wrong flow-control command. Explain why CI did not catch either problem.
Read the worked answer
flow_control: on is parsed as a boolean under YAML 1.1. In YAML 1.1, on, off, yes, no, and similar tokens are booleans. The parser produces Python True, and Jinja2 renders it as True, producing flowcontrol on. On platforms that require the full flowcontrol receive on syntax, the short form may be accepted, ignored, or applied incorrectly. The team also has mixed parsers: a YAML 1.2 parser treats on as a string, so the same file renders differently depending on tool version. This is a format-ambiguity problem, not a template typo.
ntp server True comes from the structure change. Changing ntp_servers from a list to a mapping changes what a Jinja2 loop iterates. If the template iterates the variable directly, it now iterates keys, and a boolean value can be rendered instead of an IP address. The template did not assert the variable's type or shape.
CI was green because it tested nothing meaningful. Checking that output is non-empty and contains hostname cannot detect a wrong command, a boolean leak, or a platform-incompatible syntax. There was no schema validation, no type assertion, no golden-config comparison, and no parser-version consistency check.
2. Design validation gates that would catch this class of error before any device is touched. Cover schema, parsing, rendering, and platform checks.
Read the worked answer
Gate 1: schema and type validation. Validate the inventory against a JSON Schema or equivalent before rendering. Require ntp_servers to be an array of IP addresses, flow_control to be an enum of allowed strings, and hostname to match a naming regex. Reject unknown keys. This catches the mapping change and the bare on value before Jinja2 sees it.
Gate 2: deterministic parsing and rendering. Pin one YAML loader version across tools or move inventory to JSON, which has no boolean ambiguity. Use Jinja2 StrictUndefined so missing variables fail. Render twice and require identical output; render to a structured intermediate list of commands, not just free text. Assert that no rendered line contains True, False, None, or an IP-shaped field with a non-IP value.
Gate 3: platform-aware command validation. Maintain per-platform allowed-command rules and a golden configuration for each vendor model. Check that flowcontrol receive on is emitted on platforms that require it and that the NTP lines match the intended server list. Run a diff against the last known-good rendered config and require reviewer approval for unexpected removals, especially management-plane commands.
Gate 4: connectivity guard. Block any change that removes or alters the management path unless a canary and out-of-band rollback are present. CI should fail closed on validation errors, not warn.
3. How should the team roll out the next release, including 80 new devices and a new vendor model, while making the push recoverable?
Read the worked answer
Freeze the format first. Convert the inventory to the pinned format and run the new validation gates across the existing 300 devices without pushing. Require the full fleet to render identically to the approved baseline except for intended changes. Any unexpected diff is a release blocker.
Onboard the 80 devices and the new vendor model separately. Add the new vendor's command rules and golden config before adding devices. Canary five devices on the existing model and five on the new model, with console or out-of-band access. Verify management reachability, NTP, flow control, and the new vendor's syntax after deployment, not just the render.
Make the push recoverable. Back up each device's configuration before change. Use a commit-confirm or configured revert timer where the platform supports it, or a tested configure replace rollback. Batch by site or hardware family with a pause and a health check. Stop the rollout on the first management-plane failure and restore the prior config. Keep the push script idempotent: rerunning it must not duplicate commands, and it must verify state after apply. Only after canaries and the first batch pass should the team expand to the full fleet.