NCoE Selection Test: Scope Exhaustion at 89% With Only 780 Active Devices, Plus DNS SERVFAIL on a Partner Zone

Scope Exhaustion at 89% With Only 780 Active Devices, Plus DNS SERVFAIL on a Partner Zone

Ridgeway Health's campus serves 1,400 staff devices from a single DHCP scope, 10.30.4.0/22 (1,022 usable addresses), with 8-day leases. Two Windows Server 2022 DHCP servers share the scope 80/20 with no failover relationship; relays sit on the distribution switches, and Option 82 is enabled with snooping on access ports and uplinks trusted. At 08:40 Monday the service desk receives 180 no-network tickets, and affected clients hold 169.254.x.x addresses. The scope reports 89% in use, but a scan of ARP and the NAC database finds only about 780 distinct active MACs. Most affected users are on Wi-Fi, and the wireless vendor uses per-SSID MAC randomization for staff BYOD. Separately, internal users cannot reliably resolve partner.ridgeway-exchange.net: conditional forwarders point to 172.20.8.10 and 172.20.8.11, and since Friday's firewall change that permits only UDP 53 to those servers, lookups fail intermittently with SERVFAIL while small responses still work. The firewall team says DNS is just UDP.

1. Explain the lease arithmetic and the most likely causes, then give the checks you would run and the configuration changes you would make.

Read the worked answer

Arithmetic. 10.30.4.0/22 gives 1,022 usable addresses (10.30.4.1–10.30.7.254). 89% is roughly 910 leases held, against about 780 devices actually present - so around 130 addresses are held by clients that no longer exist. With 8-day leases, a device that leaves on Monday keeps its address until the following Tuesday, and randomized Wi-Fi MACs multiply that: every reconnection with a new MAC can consume another lease for eight days. Staff BYOD, loaner laptops and imaging sessions all contribute. Note also that the 80/20 split without failover means each server reports only its own pool, so the console figure understates the combined consumption and can hide a shortage on one half.

Checks.

Export leases from both servers (Get-DhcpServerv4Lease -ScopeId ...) and join MACs against switch CAM tables and NAC sessions; count leases with no matching MAC.
Look at lease expiry distribution and the ratio of randomized (locally administered) MACs.
Verify the relay giaddr matches the scope, and check snooping/Option 82 drop counters for a rogue DHCP server or a trust misconfiguration.
Correlate the affected clients' VLAN and AP with the scope that answered.

Fixes. Shorten wireless/BYOD leases to 8-24 hours while keeping wired at 8 days; put BYOD on its own scope and VLAN; move the pair to DHCP failover (hot standby or load balance) so one console reflects both; enable conflict detection; clean stale leases; and if growth continues, plan a /21. Shortening leases raises DHCP and relay load, so measure request rates after the change.

2. Why does partner.ridgeway-exchange.net return SERVFAIL after a rule that permits UDP 53, and how would you prove and correct it?

Read the worked answer

Mechanism. DNS over UDP is limited by the response size: a classic 512 bytes, or the EDNS0 buffer size (commonly 1232). When the answer exceeds that - many records, DNSSEC material, a large CNAME chain - the server returns a response with the TC (truncated) flag set and the resolver retries over TCP 53. With TCP blocked, the retry times out and the client resolver reports SERVFAIL. Small responses still work, which is exactly the intermittent pattern observed. A second variant is fragmentation: if the oversized UDP response is fragmented and fragments are dropped, lookups also fail with SERVFAIL or timeouts.

Prove it.

dig partner.ridgeway-exchange.net @172.20.8.10 shows the tc flag or a timeout; dig +tcp at the same server succeeds.
Compare dig +bufsize=512 (forces truncation) with dig +bufsize=4096.
Check the firewall logs for denied TCP 53 sessions to 172.20.8.10/11, and confirm the rule scope covers both directions and the return path.
Check for DNS inspection/ALG or session timeouts on the firewall that might mangle or drop large UDP responses.

Correct it. Permit TCP 53 to those two servers (and back), then verify both query types from a client resolver as well as from the forwarder itself. Do not simply raise the resolver's EDNS buffer size as the fix: truncation over TCP is normal DNS behaviour and will recur with any large answer, and a larger buffer increases fragmentation risk.

3. What monitoring and documentation would you add so both faults are caught before users open tickets?

Read the worked answer

DHCP.

Alert at 70% and 85% utilization per scope, calculated across both servers and per server half, so the 80/20 split cannot mask a shortage.
Track lease request rate and NAK/decline counts per hour; a spike in declines suggests conflicts or a rogue server.
Correlate lease counts with NAC sessions weekly and report the ratio of leases to active MACs - a growing gap is the early warning for randomization and stale leases.
Alert on DHCP snooping drops and on relay unreachability per distribution switch.

DNS.

Synthetic checks against the partner zone over both UDP and TCP from an internal resolver, every few minutes, alerting on SERVFAIL.
Log and trend truncated responses; a rising rate is a leading indicator that a firewall or MTU change will break resolution.
Keep a per-forwarder health check so one dead forwarder is visible before it doubles query latency.

Documentation and change control. Record the conditional forwarders, their purpose, and the required protocols (UDP and TCP 53) in the service record. The firewall rule that caused this should never have been written as UDP-only; add a review checklist item for DNS rules that names both transports and the reason. After the incident, note the finding, the fix and the detection gap in the runbook, and schedule a quarterly scope capacity review so the next growth step is planned rather than reactive.