Large TLS transfers stall after enabling jumbo frames
An AWS EKS cluster runs Cilium 1.15 with VXLAN tunneling and kube-proxy replacement. Worker nodes use MTU 9001. Small HTTP requests between pods work, but large TLS responses and kubectl exec file transfers stall. ping -s 1400 succeeds; ping -s 2000 fails with 'message too long' or times out. cilium monitor --type drop on the source node shows drops with reason 'MTU exceeded' for packets around 8950 bytes. cilium status reports MTU 9001. The team recently enabled jumbo frames on the underlying VPC and changed the node ENI MTU to 9001, but did not change Cilium's configuration. No encryption is enabled. The application pods use standard eth0 veth interfaces, and the VPC route table has a single path with no additional encapsulation.
1. What is the root cause, and why do only large packets fail?
Read the worked answer
VXLAN adds 50 bytes of overhead for IPv4: outer IPv4 header 20, UDP 8, VXLAN 8, and the inner Ethernet frame adds 14, but the practical calculation is usually expressed as 50 bytes on top of the payload MTU. If the node ENI is 9001 and Cilium is told the MTU is 9001, the pod veth also gets 9001. A 9000-byte packet then becomes about 9050 bytes on the wire, exceeding the 9001-byte underlay. Small packets fit; large TLS records and file transfers do not. The MTU exceeded drop in cilium monitor is the giveaway. Cilium should use 8951 for this VXLAN setup, not 9001.
2. Which commands would you use to confirm the MTU mismatch and fix it? How would you verify the fix?
Read the worked answer
Confirm the actual MTUs: ip link show on the node for the ENI and cilium_host/cilium_vxlan; kubectl exec <pod> -- ip link show eth0 for the pod veth; cilium-dbg status --verbose and cilium-dbg config get mtu. Check cilium-dbg monitor --type drop while reproducing a large transfer. Fix by setting the Cilium MTU explicitly, for example Helm --set mtu=8951, then roll the Cilium DaemonSet. Do not rely on auto-detection after changing the ENI. After rollout, verify cilium-dbg status shows 8951, the pod veth is 8951, and a curl of a large object plus kubectl cp succeed. Also check that the VPC routing allows the underlay MTU path; if the path is lower, use the lower value.
3. If the team later enables WireGuard encryption on top of VXLAN, what MTU would you choose and why?
Read the worked answer
With WireGuard added, subtract both overheads. VXLAN is 50 bytes and WireGuard is about 60 bytes for IPv4, so 9001 - 50 - 60 = 8891. Set 8891 and verify with the same large-packet tests. If IPv6 is in the path, WireGuard overhead is larger, so the safe value is lower. Do not simply keep 8951 because the extra encapsulation will reintroduce the same black-hole behavior for large packets. If the platform does not need both overlays, avoid stacking them and choose native routing or one encapsulation mechanism.