OSPF ExStart and a PMTUD black hole after a DCI recabling
A regional operator runs IS-IS level 2 in its core and OSPFv2 in an aggregation domain built on seven 7250 IXR-R6 routers and two 7750 SR-12 PEs. During a data-center interconnect upgrade, all core links were re-cabled for 9,214-byte frames and the IXR interfaces were set to MTU 9214. The two PEs facing the aggregation domain still run 1,500-byte interfaces, and nobody changed them. After the change, two OSPF adjacencies on PE-1 remain in ExStart indefinitely, while every other adjacency in the same area reaches Full; hellos are exchanged continuously in both directions. Separately, a VPRN on PE-1 carries database replication between two sites: the BGP session between the sites establishes normally and small transactions succeed, but bulk transfers stall once segments approach 1.4 KB. Interface MTU changes are frozen for 48 hours by change control, and the replication SLA window closes in six hours.
1. Explain why the two PE-1 adjacencies sit in ExStart while other adjacencies in the same area reach Full, and list what you would verify before declaring MTU mismatch as the cause.
Read the worked answer
The failure is in Database Description exchange, not in hellos. A DBD packet carries an Interface MTU field, and a router that receives a DBD whose advertised MTU is larger than its own interface MTU discards the packet and the neighbor relationship never advances past ExStart. Hellos are tens of bytes and carry no MTU field, so they keep flowing and make the adjacency look healthy in a packet capture taken at hello level.
The behaviour is deterministic rather than size-dependent, which is why only the two adjacencies with one 1,500-byte end and one 9,214-byte end fail. Every other adjacency in the area is Full because both ends were changed together.
Verify before changing anything:
show router interfaceon both ends of the failing links, checking the physical port, any null or loopback binding, and the peer's view of the same link.show router ospf neighbor detailto confirm ExStart/Exchange and see retransmission counters.- Whether MTU checking has been relaxed on either side; if one side ignores the field, the adjacency can come up while the data plane stays broken, which is worse.
- That jumbo framing is actually negotiated on the IXR side, not just configured on the PE side of the link.
2. The BGP session for the replication VPRN is up but bulk transfer stalls near 1.4 KB. Explain the mechanism and what you check first.
Read the worked answer
This is a classic path-MTU-discovery black hole, and it is a different fault from the OSPF one, even though both trace back to the same MTU asymmetry.
Mechanism: the end hosts perform PMTUD on the TCP connection. Segments small enough to fit in 1,500 bytes pass unchanged. Once a segment is large enough to need fragmentation, the IP header has the don't-fragment bit set, so the 1,500-byte PE or the CE-facing path drops it and should return ICMP type 3 code 4 with the next-hop MTU. If that ICMP is filtered, rate-limited to zero, or not routed back to the sender, the sender never lowers its MSS, retransmits the same oversized segment, and the transfer stalls while the TCP handshake and small queries succeed.
Check first:
- Interface and SAP counters on PE-1 for drops and for ICMP generation.
- Any CPM filter, SAP ingress ACL or policy that discards ICMP on the VPRN path; the ICMP is generated by the router with the smaller MTU and must be permitted toward the sender.
- Whether the sender honours ICMP at all (some operating systems and appliances have PMTUD disabled or rely purely on MSS clamping).
- The actual MTU on every hop of the VPRN, including the CE-facing SAP and any GRE or IPsec tunnel in the path, since a tunnel can silently reduce the usable MTU below 1,500.
3. With interface MTU changes frozen for 48 hours and six hours of SLA left, what do you do? Justify the order of measures and say when you would escalate for an emergency change.
Read the worked answer
The freeze is on interface MTU, not on every action, so work the problem without touching MTU first.
- Restore PMTUD end to end: confirm that ICMP type 3 code 4 is generated on PE-1 and permitted back toward the replication hosts, and check it is not rate-limited into uselessness. This is a filter or policy change and carries no MTU risk.
- Clamp MSS on the CE-facing SAP of the replication VPRN if the SR OS release supports TCP MSS adjustment there; clamp to roughly 1,360 bytes so that neither direction ever emits a segment that needs the 1,500-byte path to fragment. This is the most reliable short-term fix because it removes dependence on the far-end host's PMTUD behaviour.
- If clamping is not available, steer the replication flow over the alternate path that is MTU-consistent, using a more specific static route or a policy-based route in the VPRN, accepting the extra latency and checking that the alternate path has capacity for the replication rate.
- Escalate for an emergency MTU change only if steps 1 to 3 cannot meet the window. The correct permanent fix is to align PE-1 to 9,214 bytes on the core-facing links, but that also clears the OSPF ExStart problem, so bundle both into the same change with a rollback plan rather than doing MTU work twice. Do not raise MTU on the CE-facing side without confirming the CE supports it.