Docker Certified Associate: Small requests fly, large uploads hang across nodes

• The overlay is VXLAN: each frame gains a 50-byte header (8 VXLAN + 8 UDP + 20 IP + 14 Ethernet). Docker normally derives the overlay MTU as underlay MTU − 50, but here the two rebuilt nodes told dockerd mtu: 9000, so containers scheduled there get eth0 at 9000 while the third node's containers get 1500.
• Same node: traffic goes over the local bridge; it never crosses the underlay, so the mismatch is invisible. That is why the identical call succeeds and misleads everyone into blaming the network "sometimes".
• Cross-node, jumbo → 1500 host: the sender emits 9000-byte frames with DF set. The receiving side or an intermediate switch drops them and (ideally) returns ICMP "fragmentation needed". PMTUD depends on that ICMP; when it is filtered, the sender never learns and keeps retransmitting - a classic black hole.
• Why size matters: the TCP handshake and a sub-1 KB request fit in one small segment and complete. A payload larger than one MSS needs full-size segments, which are dropped, so the write blocks until the application or socket timeout fires (~60s).
• Why the DB looks innocent: from its side a connection was opened and then simply stopped sending. pg_stat_activity shows a session waiting on bytes; no query ran, no lock was held, no slow-query log entry exists. The fault is below layer 4.

The judgement trap: raising the client timeout turns a 60-second hang into a five-minute hang. Never tune the symptom before you have measured the MTU.

Topology and configuration

docker network inspect appnet -f '{{json .Options}}'    # is com.docker.network.driver.mtu set?
docker node ls
docker service ps api --no-trunc                        # which node runs each task

Per node, from a manager or over ssh

ip -br link; ip link show <data-iface> | grep -o 'mtu [0-9]*'
grep -i mtu /etc/docker/daemon.json
docker info | grep -i 'root dir'

From inside task containers on different nodes - this is the decisive test

docker exec <api-task-on-jumbo-node> ip link show eth0        # 9000
 docker exec <api-task-on-stock-node> ip link show eth0        # 1500
docker exec <api-task> ping -M do -s 1200 <db-task-ip>        # succeeds
docker exec <api-task> ping -M do -s 1400 <db-task-ip>        # fails if black-holed

Packet-level confirmation on a mismatched node

tcpdump -ni any 'host <db-task-ip> and (tcp or icmp)'
ss -tin | grep -i retrans

You are looking for repeated retransmissions of full-size segments with no ICMP type 3 code 4 coming back.

What would rule MTU out

• eth0 MTU identical (e.g. 1450) on tasks across all three nodes and a 1400-byte DF ping succeeds cross-node → the drops are elsewhere (security group, switch ACL, or an app-level timeout).
• The stall also reproduces same-node → not encapsulation.
• tcpdump shows the receiving node ACKing everything and the application never reading → back to the app, and the DB team was right.

Do this before proposing any change; "we raised the MTU" without a before/after capture is how this bug returns in six months.

Root cause is an uneven underlay, not the app.

1. Make the fabric uniform. Either enable jumbo end-to-end (switches included) or remove "mtu": 9000 from the two rebuilt nodes. Mixed MTU inside one L2 segment is the bug; the overlay merely propagates it.
2. Pin the overlay MTU explicitly instead of inheriting whatever the node happened to be configured with:

docker network create --driver overlay \
  --opt com.docker.network.driver.mtu=1450 appnet-v2

1450 = 1500 − 50 for VXLAN. If you also use --opt encrypted, subtract the IPsec overhead as well (≈1400).

1. The MTU of an existing overlay cannot be changed in place. Create the new network and migrate services:

docker service update --network-add appnet-v2 api
docker service update --network-rm appnet api

Attaching both networks first lets tasks roll with --update-order start-first and no downtime; DNS resolution of the old name stops once the last task leaves appnet, so migrate callers together.

1. Prevent recurrence at the source: set "mtu": 1450 (or the agreed underlay value minus 50) in daemon.json through the node image or config management, and add a join-time check that compares docker exec <test> ip link show eth0 across nodes - a mismatch should fail the node build, not surface as a 3 a.m. timeout.
2. Keep PMTUD working: do not filter ICMP type 3 code 4 on the fabric, so the next mismatch degrades gracefully instead of black-holing.

Judgement call: the fastest wrong answer is --opt com.docker.network.driver.mtu=1400 everywhere with no underlay fix - it works today, silently costs ~7% throughput, and leaves the jumbo nodes as a landmine for the next workload that talks to the storage fabric directly.