Intermittent Tunnel Flaps After an ISP Maintenance Window
Following a scheduled ISP maintenance window, 40 of 120 branches report intermittent tunnel flaps. Each affected branch has two transports: fiber primary and LTE backup. The flaps occur roughly every 15 minutes on the fiber circuit; LTE takes over and traffic recovers, but the orchestrator logs repeated link-down/link-up and BFD events. One branch shows CRC errors and late collisions on the fiber interface; another shows small pings succeeding while 1400-byte pings fail. The ISP says the maintenance is complete and the circuit is clean. The business sees brief voice cuts but no major outage. You need to determine whether to escalate to the ISP, adjust SD-WAN timers, or replace hardware, without hiding a real failure.
1. Outline a triage plan to determine whether the flaps are underlay, overlay, or orchestration-induced.
Read the worked answer
First establish scope and timing: confirm that all 40 branches use the same ISP fiber platform and that flaps began at the maintenance window, not before. Compare an unaffected branch on the same ISP and an affected branch on a different ISP. Then inspect the physical layer before the overlay: interface state, CRC, late collisions, duplex, and optical levels. Next check the overlay: BFD timers, tunnel rekey logs, SLA probes, and whether the flap is a link-down event or only a BFD/probe failure. Orchestration-induced flaps would show policy pushes or configuration changes at the same cadence, so review the audit log for scheduled jobs. If the underlay link stays up but BFD drops, the problem is path quality or encapsulation, not orchestration. If the link itself drops, escalate to the ISP with interface counters and timestamps. Replace hardware only after counters point to a local port or optic and the same pattern is absent on a spare port.
2. What specific counters and logs would you collect from the affected branch and the ISP demarc, and what pattern would confirm an ISP duplex or MTU issue?
Read the worked answer
Collect interface counters for CRC, input errors, late collisions, runts, giants, and drops; duplex and speed settings; optical receive level; and carrier transitions with timestamps. Collect BFD and tunnel logs, SLA probe results for loss/latency/jitter, and ping tests at 64, 500, and 1400 bytes with DF set. At the ISP demarc, request the same physical counters and any maintenance or error logs. A duplex mismatch pattern is CRC/alignment errors plus late collisions on one side while the other side looks clean, often under load and at a regular interval. An MTU problem shows small pings succeeding and large pings failing, with tunnel establishment succeeding but data packets dropping. The 1400-byte failure at one branch strongly suggests MTU or fragmentation, while CRC plus late collisions at another points to duplex or cabling. Both are underlay faults, not orchestration faults.
3. Recommend remediation and threshold changes to prevent unnecessary failovers without masking a real outage.
Read the worked answer
Remediate the underlay first: correct duplex/speed, replace the cable or optic, and validate MTU end to end with DF pings and a temporary MSS clamp if needed. Ask the ISP to confirm the maintenance changed any port settings or MTU. Then tune SD-WAN behavior with hysteresis: increase BFD or probe failure counts slightly, add a hold-down before declaring the link down, and require multiple consecutive failed probes. Do not disable alarms or extend timers so far that a real outage is hidden. Keep LTE failover active, but add flap damping and a notification after repeated transitions in a short window. After the fix, monitor for 24 hours and compare the affected branches with control branches. If flaps persist with clean physical counters, escalate to the ISP with evidence and consider a maintenance-window rollback. Threshold changes should be documented as a temporary mitigation with a review date.