Nightly Drift Reports Show 27% Non-Compliance
NetBox is the intended source of truth for 300 access switches. A nightly job renders intended config and diffs it against running-config; the latest run shows 81 devices (27%) out of compliance. Triage finds four categories: 41 devices have interface descriptions changed by technicians, 22 have NTP and logging lines added during an incident, 12 have a stale VLAN in NetBox because a decommission was never recorded, and 6 have an ACL that was intentionally hot-fixed during a security event and never merged back. The operations lead wants the job to auto-remediate every night at 02:00. The security team wants the hot-fixed ACL preserved. Nobody currently owns NetBox data quality, and the last two audits found records that were three months stale.
1. Classify the drift and propose a policy for each category. Why is a single auto-remediate rule wrong here?
Read the worked answer
The four categories have different causes, so they need different policies.
- 41 description changes: likely benign but may encode operational knowledge. Normalize only if descriptions match a schema; otherwise flag and notify the device owner. Do not revert blindly.
- 22 NTP/logging lines: could be legitimate incident response. Compare against the approved baseline template; if the values match the baseline, fix the source record or ignore; if not, open a ticket.
- 12 stale VLANs: this is a source-of-truth quality problem, not a device problem. Never push the stale record. Reconcile the record with the device and route it to decommission review.
- 6 hot-fixed ACLs: preserve them. Create a time-boxed exception with a security owner, and merge the fix into intended config after review.
A single auto-remediate rule is wrong because drift is a symptom of either unauthorized change or bad source data. Remediating the second kind makes the network match a lie, and remediating the first kind during an active incident can cause an outage. Classification, not automation volume, is the design problem.
2. Design the remediation workflow, including safety controls for devices under active incident or change freeze.
Read the worked answer
Workflow:
- Classify each diff line into a category using rules plus a small allowlist.
- For material, non-exempt drift, open a change request with the diff and the device owner; do not push immediately.
- Auto-remediate only categories explicitly approved as safe (for example, baseline NTP), inside a window, one device at a time, with a pre-change snapshot and post-check.
- Use the same transaction semantics as the deployment pipeline: confirmed-commit where the platform supports it, checkpoint/rollback otherwise.
- Require a second approver for ACL, routing, and any device currently in an incident.
- Write an audit record for every action: diff, actor, result, timestamp.
Safety controls: a per-device freeze flag that skips remediation and opens a ticket; a global change-freeze calendar; a blast-radius cap (for example, no more than 10 devices per hour); and an automatic stop if post-check failures exceed a threshold. A nightly 02:00 push during an outage is exactly the wrong behavior.
3. What governance and metrics would you put around NetBox so the drift rate becomes meaningful?
Read the worked answer
Assign a named owner for NetBox data quality in network operations, not in the automation team. Add a weekly review of stale records and a decommission workflow that starts in NetBox, so removal from the source of truth is the first step rather than an afterthought.
Automated checks:
- Every device in NetBox must exist in the inventory and vice versa.
- Records older than 90 days without review are flagged.
- Rendered config must be reproducible from the source of truth with no manual overrides.
Metrics: drift by category, mean time to reconcile, and false-positive rate of the classifier. A drift rate that falls from 27% to 5% because records were fixed is different from one that falls because the job stopped reporting. Publish both the drift count and the source-of-truth accuracy. Set an SLO: material drift reconciled within 48 hours; benign drift within the next maintenance window.