400 Wired Endpoints Fall to the Guest VLAN After a PKI and RADIUS Certificate Change
Halden Manufacturing runs 802.1X on 42 access switches for about 1,200 endpoints. Wired endpoints use EAP-TLS with machine certificates from a 3-year internal PKI; printers and building-management controllers use MAB; successful authentication assigns a dynamic VLAN. The FreeRADIUS 3.2 pair had its EAP server certificate reissued on Friday, and the PKI team also published a new intermediate CA and root as part of a planned migration. On Monday at 05:30 about 400 wired devices fail to authenticate. Some supplicants log TLS alert: unknown CA, others log certificate expired; several switches show ports landing in VLAN 999, which permits only DHCP, DNS and the remediation portal. Switch templates differ: the older template uses authentication order dot1x mab with mab in the method list; the newer template shipped last quarter omitted mab from the method list. One RADIUS node shows 11 minutes of clock drift after a hypervisor migration. The helpdesk has 400 tickets and is asking whether to disable 802.1X globally.
1. Rank the failure signatures by blast radius and explain how you would separate them using RADIUS and supplicant evidence.
Read the worked answer
Three distinct causes are present, and they must be separated before any fix, because the remedies touch different systems.
- Expired RADIUS EAP server certificate - every EAP-TLS supplicant that reaches that node fails, regardless of client state. This is the broadest and most deterministic signature, and the fastest to fix. Check with
openssl x509 -in server.crt -noout -dateson both nodes and withfreeradius -Xon a test request. - Clients that do not trust the new root/intermediate - a subset, correlated with devices that have not received the new root via GPO/Intune or that were offline during the push. Signature:
unknown CA, and the same device succeeds against a test SSID/port using the old chain. - NTP drift on one node - intermittent, roughly half of attempts, and only when the load balancer lands on the skewed node. Signature:
certificate expiredornot yet validon the server side despite correct dates, pluschronyc trackingshowing a large offset. Note thatcertificate expiredcan also come from a client with a dead CMOS battery, so compare the failing client's clock as well.
Order of work: fix the server certificate and the clock first - both are server-side, reversible and remove the largest failure class within minutes. Then distribute the root/intermediate and re-test a sample from each OU. Use accounting Start records grouped by NAS and by RADIUS node to prove the split, and keep a known-good laptop as a control for each change.
2. Why did the affected ports end in VLAN 999, and what are the security implications of simply enabling MAB everywhere as a quick fix?
Read the worked answer
Why VLAN 999. On the newer template, dot1x is the only method in the method list, so when EAP-TLS fails there is no MAB attempt and the port falls to the authentication-failure behaviour, which is configured as the restricted VLAN 999. On the older template, mab is in the method list and the headless devices still authenticate. So the split between working and broken switches matches the template version, not the endpoint type - a useful confirmation that the certificate problem and the VLAN problem are separate faults.
Immediate fix. Add mab back to the method list on the newer template, push it with an automation run limited to the affected switches, and force re-authentication (CoA or a port bounce) instead of waiting for the reauthentication timer. Verify with show authentication sessions interface and RADIUS accounting that devices land in their intended VLANs.
Security implications. MAB authenticates a MAC address, which any attacker can spoof, so enabling it globally silently downgrades every failed EAP-TLS port to a weak credential and effectively defeats 802.1X for anyone who can observe a MAC. MAB should be enabled only on ports profiled for headless devices, ideally combined with profiling and a restrictive ACL or a dedicated VLAN. Also confirm the failure behaviour is fail-closed (VLAN 999 with a minimal ACL is reasonable) and that 999 cannot reach production subnets. If business pressure demands a blanket change, time-box it, log it as a risk acceptance, and revert within days.
3. What would you change so the next certificate or PKI change does not produce this outcome?
Read the worked answer
The failure was a sequencing and observability problem, not a cryptography problem, so the controls should target both.
Sequencing and dependency checks. Distribute the new root and intermediate to all endpoints (GPO/Intune/MDM) and confirm coverage before the RADIUS server switches chains. Change records for the RADIUS server certificate should name its dependants: supplicant trust stores, the load balancer health check, monitoring probes and any appliance that validates the chain. Add a pre-change checklist item asking whether any client must trust something new.
Technical guardrails.
- Certificate expiry monitoring at 30/14/7 days for every server certificate and CA, alerting to the team that owns the renewal.
- NTP monitoring with an alert at more than 1 second offset on any RADIUS node; certificate validity checks are meaningless without it.
- A synthetic 802.1X test account and supplicant probing each RADIUS node every few minutes, alerting on failure or on a node imbalance.
- A canary ring: 5% of endpoints, one per switch template, authenticate against the new configuration first; only then roll forward.
- Template tests in CI that assert
mabis present where required, so a template change cannot silently alter failure behaviour.
People and process. Rehearse the PKI migration on a lab switch with one of each template, run the change in a low-traffic window with a named rollback, and give the helpdesk a decision tree that distinguishes 'expired server certificate' from 'unknown CA' before anyone proposes disabling 802.1X globally.