Morning APIPA Outage at a 300-Seat Call Centre
A 300-seat call centre runs one flat data VLAN, 10.10.10.0/24, with DHCP and DNS on a Windows server at 10.10.10.5 in the same VLAN, lease duration 8 days. Desk hard phones sit in VLAN 20 (10.10.20.0/24) and are served by the same DHCP server via ip helper-address on the VLAN 20 SVI; phones are stable. Every morning at about 07:45, between 40 and 60 agents receive 169.254.x.x addresses and cannot log in; rebooting fixes some of them. The DHCP log records 'no free leases' against the 10.10.10.0 scope only. Internet access is PAT to a single public address on the edge router, whose NAT table limit is 32,768 entries; peak concurrent sessions measured at 30,000. Agents also report that an internal application published as crm.example.com resolves to the public IP and performs poorly compared with the same app reached by IP.
1. Diagnose the morning APIPA failures and justify the fix with the arithmetic.
Read the worked answer
10.10.10.0/24 provides 254 usable addresses for roughly 300 devices, so the scope is too small by design. The log line 'no free leases' on that scope, combined with healthy phones on VLAN 20, rules out a relay or server fault - the server and the ip helper-address on the VLAN 20 SVI are working.
The 8-day lease compounds it: hot-desked machines, spare PCs and laptops that leave the building keep their addresses for days, so the pool drains and refills slowly. At 07:45 everyone arrives at once, the last leases are taken, and the remainder fall back to 169.254.x.x. Rebooting only helps when a client happens to win a race against an expiring lease.
Fix: move the data VLAN to 10.10.10.0/23 (510 usable) or split it, reduce the lease to 8–12 hours, purge stale leases, and alert at 80 % scope utilisation.
2. The NAT table is close to its limit. Explain why that matters and rank the mitigations.
Read the worked answer
PAT rewrites the source port, so every concurrent flow consumes a translation entry; 30,000 against 32,768 leaves under 10 % headroom, so a peak will hit the ceiling and new sessions - including softphone registrations - will fail or hang.
In order of cost and risk:
- Reduce idle timeouts for short-lived HTTP flows so entries free faster. Verify SIP registration/re-INVITE timers before shortening voice timeouts, or calls will drop.
- Add a second public IP to a NAT pool, spreading ports across two addresses and roughly doubling capacity.
- Hunt for applications opening excessive short-lived connections - a real cause more often than expected.
- Give the softphone registrar a static one-to-one NAT if embedded IPs in SIP/SDP break under PAT.
Monitor show ip nat statistics and alert at 80 % so the next peak is not a surprise.
3. crm.example.com resolves to the public address for internal users. What is happening and what is the better fix?
Read the worked answer
Internal clients send traffic to the public IP, so packets go to the edge router, must be translated back to the internal server, and the replies translated again - hairpin (reflection) NAT. Many routers do not support it on the same interface, and where it works it consumes scarce NAT entries and hides real client addresses from the application logs.
Better fix: split-horizon DNS. The internal DNS zone returns the private address (for example 10.10.10.20) while the public zone keeps the public IP for external users. Traffic stays on the LAN, the NAT table is not involved, and source addresses are preserved for logging and troubleshooting.
Hairpin NAT is the fallback only if DNS cannot be split - for instance if clients use external resolvers, which should itself be corrected.