ZTP and Telemetry Rollout for 60 Switches with Configuration Drift
A service provider is deploying 60 QFX switches across three pods. The current manual process takes two engineers three days per pod, and 11% of devices drift within 30 days, mostly NTP, syslog, and VLAN changes. The team knows Junos PyEZ but has no full orchestration platform. Requirements are zero-touch provisioning from DHCP, a golden configuration with per-device variables (hostname, management IP, loopback, ASN), commit-confirmed rollback, streaming telemetry every 10 seconds for interface counters and 60 seconds for BGP, and a configuration audit every 24 hours. The DHCP server can return option 43 and a boot-file name. The operations manager wants a design that can be implemented with Python, Git, and existing Linux servers, not a new vendor platform.
1. Design the ZTP workflow from DHCP through a committed configuration, including safety rollback.
Read the worked answer
Configure DHCP to return option 43 with the Junos image URL and a configuration URL, or use the boot-file name to point to a bootstrap script. On first boot, the switch obtains the image, installs it, reboots, and fetches a per-device configuration. Generate that configuration from a source of truth such as YAML or NetBox using Jinja2, with hostname, management IP, loopback, and ASN as variables; store the rendered configs and templates in Git. The device should apply the config with commit confirmed for 10 minutes and call a confirmation script or webhook after it establishes management connectivity. If confirmation does not arrive, Junos rolls back automatically. Use a serial console or out-of-band management for recovery. Do not embed shared secrets in the config; use a bootstrap credential that is rotated after provisioning. This workflow stays within Python, Git, and Linux HTTP/DHCP services.
2. Which telemetry transport and data model should be used for 10-second interface counters and 60-second BGP statistics?
Read the worked answer
Use gNMI over gRPC with OpenConfig models, secured with TLS, on the standard gRPC port. Subscribe to OpenConfig interfaces for 10-second counter samples and OpenConfig BGP for 60-second neighbor and prefix statistics. gNMI gives structured, self-describing data that is easier to store, diff, and alert on than SNMP polling, and it supports subscriptions rather than repeated polls. The alternative, Junos Telemetry Interface over UDP, has lower overhead but less standardized decoding and weaker subscription control. At 60 devices, the collector load is modest, but design for per-device export rates and use a redundant collector pair. Ensure the telemetry path is separate from the in-band management network where possible, and test that a collector outage does not affect forwarding. Keep a fallback SNMP poll for devices that fail to establish gNMI.
3. How would you detect and remediate configuration drift every 24 hours using only PyEZ, Git, and cron?
Read the worked answer
Run a nightly Python job that uses PyEZ to retrieve the running configuration in a normalized text form, for example show configuration | display set. Render the golden configuration from the same Git-stored templates and per-device variables, then diff the two after removing expected volatile lines such as timestamps and commit history. Record the diff and the device's current commit in Git or a log repository. Classify drift by field: NTP, syslog, and approved VLAN changes can be auto-remediated; routing policy, ASN, and authentication changes should open a ticket. For auto-remediation, load the golden config with replace semantics, commit confirmed for 10 minutes, verify management reachability, and confirm; otherwise roll back. Run the job from cron on a redundant Linux host, alert on failures, and review the drift report weekly. This gives an audit trail without a full orchestration platform.