Failover, a Flapping Mobility Tunnel, and Guest DHCP at the Remote Site

What HA SSO preserves

• The active and standby keep a synchronized copy of configuration and AP state, and the APs stay connected because the redundancy path preserves the identity the APs registered with. That is why APs stayed up.
• The RP/management identity is preserved so APs and the network see the same controller identity after switchover.

What it does not give you for free

• Client state synchronization depth depends on release. In particular, confirm whether PMK/PMKID caching is synchronized for your code version. If it is not, every 802.1X client performs a full EAP exchange, which is exactly the symptom described, and the fix is design (fast transition, session resumption on ISE, or accept the re-auth and size ISE for the burst).
• Any session that depends on an external peer needs that peer reachable at the active unit's address.

Checks before the next test

1. Redundancy ports and redundancy management (RMI) link: RMI and management must be on separate subnets, each unit's RMI in the same subnet as its peer's, with a working default gateway. A shared or congested RMI path inflates switchover time.
2. Confirm the standby is fully synced and hot; look for sync errors and config drift between units, and confirm identical image versions.
3. Confirm the mobility peer on the second site points at the primary pair's management address, not at the standby's, and that the peer's own configuration references the correct address.
4. Check MTU and packet loss on the path to the peer: mobility control keepalives that are dropped show up as a flapping tunnel.
5. Time the switchover with a controlled trigger and capture on the peer to see whether the tunnel flaps or is administratively rebuilt.

Rough target A properly built HA SSO pair should converge well under the 2 seconds measured; treat 2 seconds as evidence of a problem to find, not as normal.

Configuration prerequisites

• Identical mobility group name on all controllers in the group. A name mismatch silently prevents peer relationships.
• Each controller has a unique virtual interface IP; duplicate virtual interfaces cause the peer relationship to fail or behave erratically.
• The peer address configured on each controller is the other controller's active management address. In an HA SSO pair, do not point the remote site at the standby address.
• Symmetric mobility tunneling should be consistent on both sides; mixed settings are a classic cause of "the tunnel is up but clients lose traffic on handoff".

Firewall rules

• Mobility control UDP 16667 and mobility data UDP 16666 between the primary pair and the remote controller.
• If the mobility tunnel is configured to use EoIP for data, permit IP protocol 97 instead of, or in addition to, UDP 16666. A firewall that blocks protocol 97 while permitting 16667 gives you a control plane that looks perfectly healthy and a data plane that is dead.
• Bidirectional, stateful-friendly rules; asymmetric routing across the firewall will break handoffs intermittently.

Verification beyond "up"

• Confirm the peer shows a current heartbeat and that the tunnel has not flapped recently; check counters rather than status text.
• Move one test client across the boundary and confirm in the client detail that the foreign/anchor relationship is as designed, and that traffic actually flows after the handoff in both directions.
• Watch the mobility data counters increment during the roam; a handoff with zero data counter movement means the client moved but its data path did not.
• Test during a maintenance window with an actual long-lived session (a ping plus an authenticated 802.1X client), not with association alone.

Leading hypothesis The mobility control tunnel is up but the data tunnel is not passing. Association to an anchored WLAN requires the controller to reach the anchor, so control on UDP 16667 is working; DHCP is carried in the data path to the anchor's guest VLAN, and if the firewall now blocks UDP 16666 (or IP protocol 97 if EoIP is in use), the DHCP Discover never reaches the anchor and the client sits with no address while the tunnel still displays as up. This matches "associates but no IP after a firewall change" exactly.

How to prove it

1. Compare the anchor and foreign client detail for a failing second-site client: it will show as anchored, with the anchor relationship up.
2. Capture on the anchor's guest VLAN service interface while a client tries to get an address: no Discover arrives.
3. Capture on the foreign controller's mobility interface: Discover leaves and nothing returns, or nothing leaves at all if the data path is being dropped before it is tunneled.
4. Check the firewall logs for denies on UDP 16666 or protocol 97 between the second site controller and the DMZ anchor during the test window.

Other things to rule out before blaming the firewall

• The DMZ DHCP scope for the second site may be exhausted or missing a relay/helper for the anchor's guest VLAN.
• The anchor may have failed over to a secondary anchor whose guest VLAN has no DHCP relay.
• The guest WLAN on the second site may be pointing at a different anchor than expected.
• A VLAN mismatch on the DMZ switch after the firewall change (native VLAN or trunk pruning) would produce the same symptom.

Fix and verification Restore bidirectional UDP 16666/16667 plus protocol 97 as applicable, then verify with a real client receiving an address and completing portal login, not with a tunnel status check.