Control Connection Flaps After a Circuit Upgrade
A logistics company upgraded 12 of its 60 branches from 100 Mbps to 1 Gbps fiber. Within a week, those 12 branches show control connections flapping every 20–40 seconds, OMP sessions reset, and some TLOCs drop, although IPsec data tunnels and BFD remain up. Voice calls unnecessarily fail over to LTE. The new ISP uses carrier-grade NAT and does not provide a static public IP; the old ISP did. The vBond is public, while vSmart and vManage sit in the data center behind a firewall. The new circuit has a 1400-byte MTU; the old circuit was 1500. CPU and memory are normal, latency to controllers is 18 ms, and certificates have not expired. vManage reports control connection down but tunnel up. The team has a change freeze in 48 hours and needs a focused plan.
1. What differential diagnosis and priority checks would you run first?
Read the worked answer
Separate control plane from data plane. BFD up proves the IPsec data tunnel and BFD packets are working; control connections are DTLS/TLS to vBond, vSmart, and vManage outside the IPsec tunnel. The 12 upgraded branches share three new variables: ISP, CGNAT, and MTU 1400. Prioritize checks that match all three.
First, check control connection history for the disconnect reason. Second, verify vBond reachability and that its public IP/DNS is correct. Third, confirm outbound UDP 12346 and TCP 443/8443 are not blocked and that firewall state timeouts are longer than the keepalive. Fourth, compare the CGNAT mapping timeout with the NAT keepalive; a 20–40 second flap strongly suggests mapping expiry or DTLS keepalive failure. Fifth, test MTU with DF-bit pings at 1400 and check for fragmentation drops. Sixth, verify certificate validity, clock, and duplicate system IP/site ID. Use show sdwan control connections-history, show sdwan control connections, show interface, traceroute, and logs.
2. Why can BFD and IPsec tunnels stay up while control connections flap?
Read the worked answer
BFD runs inside the IPsec tunnel and uses small, frequent packets. Control connections use DTLS/TLS outside the tunnel and include larger handshake and certificate exchanges. A path that passes small BFD packets can still fail on a large control packet if MTU/fragmentation is wrong, or fail on return traffic if a CGNAT mapping expires. The control connection then reconnects, OMP resets, and TLOCs may drop even though the data tunnel never went down.
The voice failover to LTE is a downstream policy symptom, not the root cause. OMP resets make the branch appear to lose paths, so AAR selects LTE. Do not chase voice policy until the control connection is stable. Also check whether the data center firewall is doing stateful inspection on the control connections; a short UDP timeout can mimic CGNAT expiry.
3. What remediation and validation plan would you use before the change freeze?
Read the worked answer
Set the WAN interface MTU to 1400 and adjust TCP MSS (for example 1360) so control and management sessions do not rely on fragmented large packets. Lower the NAT keepalive below the CGNAT mapping timeout, or request a static public IP/dedicated NAT from the ISP; CGNAT is the likely trigger because the old ISP provided a static address. Confirm the data-center firewall allows UDP 12346 and TCP 443/8443 and has sufficient state timeout. Verify certificates and time again after any address change.
Validate on one branch: control connections stable for at least 24 hours, OMP routes present, BFD stable, and voice no longer failing to LTE. Repeat on a second branch with the same CGNAT type. If the ISP cannot guarantee mapping behavior, move that branch to a business circuit with a public IP or tunnel over a stable overlay. Do not make a mass change during the freeze; apply the proven fix to the 12 branches as a controlled change with rollback.