Retail WAN Outages: Local Mode Versus FlexConnect and EWC
A 240-store retail chain runs three to six Catalyst APs per store on a local access switch, a 100/20 Mbps DIA circuit, and an IPsec tunnel to a data center with 35-60 ms RTT and roughly three unplanned WAN outages per month lasting 5-20 minutes. Two Catalyst 9800-80 controllers in the data center run HA SSO and currently terminate every AP in local mode; all WLANs are centrally switched, and every store POS subnet is served by DHCP from the data center. Last quarter three outages took point-of-sale offline, and guest traffic is saturating the IPsec tunnels. Separately, after the last controller image upgrade, APs at 14 stores never rejoined.
1. Recommend a switching and authentication model for POS, corporate, and guest, and name the configuration elements each one needs.
Read the worked answer
The requirement is that POS survives a WAN outage, so nothing POS-critical can depend on the tunnel.
- POS: FlexConnect with local switching. The store switch trunks to the AP, per-WLAN VLAN mapping places POS traffic on the local POS VLAN, DHCP/DNS come from the store switch or local server, and ACLs are applied at the AP. Authentication has to survive too: centralized 802.1X cannot authenticate new clients when the RADIUS path is down, so use FlexConnect local authentication (local user list or a local RADIUS instance) or PSK with MAC filtering, and document the trade-off. Already-authenticated clients can keep running on a cached PMK, but any re-auth fails.
- Corporate: FlexConnect local switching with centralized 802.1X where policy matters, accepting that new authentications fail during an outage, or extend local authentication to it if the business will not accept that.
- Guest: local switching with direct internet breakout at the store. Trombone-tunneling guest to the data center is what is filling the IPsec tunnels; breakout frees the tunnel and removes the WAN from the guest path. Also confirm the AP VLAN support is enabled on the 9800 and that each AP's switch port is a trunk with the correct native and allowed VLANs.
2. Compare keeping the two data-center 9800-80s in HA SSO against deploying a Catalyst 9000 embedded wireless controller per store. Which do you pick and on what criteria?
Read the worked answer
Data-center HA SSO: one policy domain, one place to change configuration, mature HA, and 1,200 APs is well within a 9800-80's scale. The cost is that the WAN is in the path of every centralized function, and with three outages a month the branch experience is dictated by the ISP.
Per-store embedded controller: full local survivability including authentication and policy, no dependence on WAN for data or control, and it rides on hardware the store already needs. The cost is 240 management points, per-site config drift, and a refresh cycle tied to the switch.
Decision criteria: WAN reliability and SLA, whether any store application is latency-sensitive (POS is), operational maturity to automate 240 sites, and whether the business will fund switch refresh. With this WAN and a POS that must stay up, I would go local: FlexConnect first because it is a config change on existing hardware, and embedded controllers where a store needs a full local feature set. Whichever path, drive configuration from templates and a controller-management platform, not by hand, and verify whether your embedded-controller platform and release support HA before promising it.
3. APs at 14 stores never rejoined after the controller image upgrade. Give a prioritized troubleshooting sequence and the likely culprits.
Read the worked answer
Work outward from discovery to join to image to license:
- Discovery: are the APs getting the correct controller address? Check DHCP option 43, DNS resolution of CISCO-CAPWAP-CONTROLLER, and whether the management IP changed with the upgrade. Stale option 43 strings are the classic cause of a store-wide non-join.
- Reachability: CAPWAP UDP 5246/5247 open through the store firewall and NAT, MTU/fragmentation not breaking DTLS, and the AP landing in the right VLAN.
- Time and certificates: NTP skew breaks DTLS; check the AP's certificate type against the controller's trust settings (MIC versus SSC versus LSC) if the controller was hardened during the change window.
- Image mismatch: the AP must download the matching image over a 100/20 Mbps circuit. Look for stuck image download or predownload, and consider predownloading and priming APs at a pilot store.
- Regulatory domain and country code mismatch, which blocks the radio.
- Licensing: AP count or entitlement exceeded puts APs in a not-joined state. Given 14 stores failed and 226 did not, start with what those 14 share: likely an option 43 or firewall template difference, not the controller.