Branch HTTPS stalls and OSPF flaps over an IPsec tunnel

The pattern points to a path-MTU black hole, not a policy deny and not an OSPF-only fault.

Reasoning

• A policy deny would block the SYN as well; the handshake completing means policy and routing are correct for small packets.
• Small packets (ping default 56 bytes, TCP SYN/SYN-ACK) fit any plausible MTU. Anything carrying a full 1500-byte segment with DF set is dropped somewhere on the PPPoE/IPsec path, and returns nothing.
• OSPF flapping between Full and ExStart is the classic signature of an MTU mismatch: hello packets are small and get through, database description packets are large and do not.

Three checks

1. Size sweep with DF: execute ping-options df-bit yes, then execute ping-options data-size 1300 and 1500 to 10.50.8.20. Success at 1300 and failure at 1500 isolates the break to MTU.
2. Compare MTUs: get system interface (or diagnose netlink interface list) for the PPPoE WAN, HQ-VPN, and HQ's side, and check whether any MSS clamping exists on the user policies.
3. Capture for ICMP type 3 code 4 (fragmentation needed) in both directions. If PMTUD is being used but the ICMP is filtered, the sender never learns the smaller MTU.

Run diagnose debug flow once with a filter on 10.50.8.20 to formally exclude a policy drop before changing anything.

Values

• PPPoE WAN: 1492.
• ESP overhead for AES/SHA-style tunnels is roughly 50 to 73 bytes, so 1492 minus overhead lands near 1419. Setting the tunnel MTU to 1400 leaves margin.
• TCP MSS = tunnel MTU minus 40 (IPv4 + TCP header) = 1360.

Changes at the branch

1. Set HQ-VPN MTU to 1400.
2. Clamp MSS to 1360 on the policies that carry user traffic to HQ (tcp-mss-sender / tcp-mss-receiver, or the interface MSS setting if the build supports it). Clamping at both ends is safer for return traffic.
3. Ensure ICMP type 3 code 4 is permitted in both directions so PMTUD still works for non-TCP traffic.

Changes at HQ Set HQ's tunnel interface to the same 1400 MTU. This is the important part: matching MTUs fixes the OSPF adjacency at the source rather than hiding it with set mtu-ignore enable on the OSPF interface. Use mtu-ignore only as a temporary workaround while the MTU change is scheduled.

Why not MSS clamping alone MSS clamping only repairs TCP. OSPF, UDP/voice, and any DF-set UDP traffic would still break, and the OSPF flaps would continue. Fix the MTU first, then clamp MSS for TCP performance.

• Handshake: SYN and SYN-ACK are tens of bytes. They traverse PPPoE and the ESP encapsulation without exceeding any link MTU, so the three-way handshake completes and the session is created on both FortiGates with correct NAT and policy state.
• Data phase: the client and server negotiate an MSS based on their local interface MTUs. If the server believes 1500 is available, it sends full-size segments with DF set. Those segments exceed the PPPoE/tunnel path MTU and are dropped at the link that cannot fragment them.
• Why no recovery: PMTUD depends on receiving ICMP fragmentation-needed messages. If those are filtered, the sender keeps retransmitting the same too-large segment until the connection stalls. The ~1.4 KB stop point is the tell: exactly one maximum-size segment gets through or is acknowledged, then nothing.
• OSPF: hellos are small and keep the neighbour relationship alive momentarily, but database description and link-state update packets are large. They are dropped for the same reason, the adjacency falls back to ExStart, retries, and eventually reaches Full again when smaller packets succeed. The flap is a symptom of the same MTU fault, not a second problem.