Automated VPN change blows up: blast radius and BGP containment
An enterprise customer has 320 sites on an L3VPN delivered by a pair of 7750 SR PEs, with provisioning automated through Nokia NSP. A change window added 40 sites using a template; the route policy that constrained import and export was omitted. Within nine minutes 1,200 prefixes leaked from the VPN into the global routing table, a default route was pushed to all 320 CPEs, and the outage lasted 42 minutes. The review found no pre-change validation, the intent was applied to all 40 nodes in one transaction, and rollback was manual per node. The lab has six nodes, and NSP Workflow Manager is licensed but unused. The customer wants a design that makes a repeat unable to cause a 42-minute outage, plus evidence for auditors.
1. Redesign the change pipeline so a bad template cannot reach 40 nodes at once. What are the gates and the abort criteria?
Read the worked answer
Make the policy mandatory in the template. Model the route policy as a required object in the intent schema, not an optional field. If it is absent, rendering fails before anything is sent to a router.
Render, diff, validate. Produce the intended config, diff it against the running config, and run automated checks: VRF present, RT import/export present and correct, route policy attached in both directions, max-prefix set, no default-originate unless explicitly requested.
Canary and batches. Apply to two sites, soak for a defined period, verify BGP sessions, prefix counts per VRF, RTs and policy counters, then move in batches of five to ten with the same soak. Never 40 in one transaction.
Abort criteria. Session flap, prefix count outside an expected band, any prefix appearing in the global table, or post-check timeout - abort and roll back immediately.
Automated rollback. Snapshot the pre-change config, pre-test the rollback workflow in the six-node lab, and make rollback a one-click workflow with a target of under 5 minutes. Manual per-node rollback is what turned a bad template into 42 minutes.
2. Which routing safeguards would have contained the leak and the default route? Be specific about where they sit.
Read the worked answer
Per-VRF max-prefix on the PEs. Set a warning at about 80% of the expected prefix count and a shutdown at 120%. A 1,200-prefix leak into a VRF sized for a few hundred would have tripped the limit and protected the CPEs.
Import and export route policies as mandatory objects. The export policy from the VRF should only tag and advertise the customer's prefixes with the correct RT; the import policy should only accept the matching RTs. The omitted policy is exactly what allowed the leak.
Default-route guard. Default-originate toward CPEs should require an explicit intent flag and a separate approval. A default route reaching 320 CPEs is a design decision, not a side effect of a template.
eBGP edge hygiene. Inbound prefix filters built from IRR/RPKI, RPKI ROV where the customer's prefixes are signed, TTL security, and maximum-prefix on eBGP sessions. These do not stop an iBGP leak, but they stop a leaked route being accepted from outside.
Global table monitoring. Alert on any prefix in the global table that matches the customer's address space. That is the detection control that shortens the outage even when prevention fails.
3. What evidence package would satisfy the auditors, and how do you prove the new process works?
Read the worked answer
Per-change record. Intent version, rendered configuration hash, the diff, the validation results, the approver, the canary and batch timings, the post-check output, and the rollback record if used. Store it immutably - NSP audit logs exported to write-once storage, with the intent itself versioned in git.
Traceability. Link each deployed config to the intent version that produced it, so an auditor can answer 'why does this router have this policy' without reading a CLI history.
Control evidence. Show that the pre-change gate is enforced, not advisory: a failed validation must block the workflow. Show max-prefix values per VRF and the alert thresholds. Show that rollback is automated and timed.
Prove it with a game day. Once a quarter, deliberately apply a bad template in the lab and in a controlled production canary, and measure detection, abort and rollback times against the 5-minute target. Record the exercise. An auditor will accept a tested control; an untested workflow with a licence sitting unused is exactly what failed here.