Cilium Native Routing Uses 9000-byte Pod MTU over a 1500-byte BGP Path
A platform team runs Kubernetes 1.27 with Cilium 1.14 in native routing mode on 12 worker nodes. Each node has a 25 GbE data NIC with MTU 9000 and a 1 GbE management NIC with MTU 1500. Cilium auto-detects MTU 9000 from the data NIC and sets pod MTU 9000. The Cilium BGP session advertises pod CIDRs over the management network, so the ToR installs pod routes with a next hop reachable via the 1500-byte management path. Same-node pod-to-pod iperf3 reaches 20 Gbps. Cross-node pod-to-pod with 1400-byte messages reaches 9 Gbps, but a 64 MB TLS upload stalls near 1.2 Gbps and sometimes resets. Small cilium-health probes show 0% loss. The ToR data VLAN supports MTU 9000. The team wants to keep native routing and avoid encapsulation.
1. Identify the MTU inconsistency and explain why small probes pass while bulk TLS stalls.
Read the worked answer
The inconsistency is that Cilium sets the pod/device MTU to 9000, but the BGP next hop for remote pod CIDRs is reachable over the 1500-byte management network. The data NIC's 9000-byte MTU does not help if the route installed by BGP points at the management path. Small probes fit; a 64 MB TLS upload sends full-size segments that exceed the 1500-byte path and are dropped, causing stalls and resets. Same-node transfers stay on the local bridge and are unaffected.
Confirm with cilium status --verbose for the detected MTU, ip route get <remote-pod-IP> to see the egress interface, ip link MTU on both NICs, and ping -M do -s 1472 versus -s 8972 across nodes.
2. What should be changed first, and what data would you collect to prove the fix?
Read the worked answer
Fix either the route or the MTU. Preferred: peer Cilium BGP over the 9000-byte data network and make the ToR install pod routes with a data-network next hop. Then keep pod MTU 9000 and native routing. If BGP must stay on management, set Cilium's MTU to 1500 and enable TCP MSS clamping so pods do not emit 9000-byte packets onto a 1500-byte path. Do not rely on fragmentation; it hides the mismatch and hurts performance. Validate both small probes and bulk TLS after the change, and check that the ToR has no stale routes from the old next hop.
3. Recommend a target design that preserves native routing and prevents recurrence.
Read the worked answer
Keep native routing, but make the underlay consistent: BGP over the data NIC, ToR MTU 9000 on the data VLAN, Cilium device and pod MTU 9000, and no management-network next hop for pod CIDRs. If management BGP is a hard requirement, standardize on MTU 1500 and MSS clamping cluster-wide rather than mixing 9000 and 1500. Add a preflight check that compares cilium status MTU, node interface MTU, and BGP next-hop interface MTU. Test with DF-bit pings and a multi-megabyte TLS transfer before scaling.