Overlay MTU blackhole and an audit requirement for client IPs
A legacy billing application is being migrated to Docker EE 3.0 Swarm. It has two components: a frontend billing-web over HTTP and a backend billing-ledger that binds to UDP port 514 and must see the original client IP address for audit. The cluster nodes are on an AWS VPC whose path MTU is 1500, and the Docker overlay network uses the default 1500 MTU. A proof-of-concept service using the overlay works for small HTTP requests, but large POST requests intermittently hang. Inside containers, ping -M do -s 1472 succeeds, while 1473 fails. The security team requires that ledger traffic be encrypted in transit and that the ledger not be reachable from outside the swarm. The team proposes attaching billing-ledger to docker_gwbridge and publishing port 514.
1. Diagnose the intermittent hangs and specify the network change.
Read the worked answer
The underlay path MTU is 1500, proven by 1472-byte payload success and 1473-byte failure. VXLAN adds 50 bytes of overhead, so an overlay configured at 1500 sends 1550-byte frames that cannot traverse the underlay. Large POSTs depend on fragmentation or path MTU discovery, and PMTUD blackholes cause the hangs; small requests fit. Create or recreate the overlay with --opt com.docker.network.driver.mtu=1450. For existing services, create a new network, connect the services, and remove the old one during a maintenance window. Validate inside containers with ping -M do -s 1422 (1450 minus 28 bytes of IP and ICMP headers); 1423 should fail. Check docker network inspect to confirm the MTU option is present on every endpoint before declaring the fix complete.
2. How should the ledger be exposed to meet both the audit and security requirements?
Read the worked answer
Do not publish UDP 514 through the routing mesh: that exposes the ledger on every swarm node and source-NATs the client IP, defeating the audit requirement. Attach billing-ledger only to an encrypted overlay network, for example docker network create --opt encrypted --driver overlay ledger-net, and let billing-web reach it by service name. Preserve client identity at the application layer: have billing-web pass the original client IP in a signed header or PROXY protocol payload, and have the ledger log that field. If the ledger must receive raw syslog directly, use a dedicated relay sidecar at the edge that writes to the overlay, or bind a host-mode port only on an internal interface protected by a security group. That is an explicit exception, not a default. Never attach the ledger to docker_gwbridge for external access.
3. What commands and observations prove encryption and MTU are working?
Read the worked answer
Validation should cover both controls. For encryption, inspect the network options with docker network inspect -f '{{json .Options}}' ledger-net and confirm encrypted is true. Capture underlay traffic with tcpdump -n -i eth0 esp while the ledger exchanges messages; Swarm overlay encryption uses IPsec ESP, so payloads should not appear in clear text. For MTU, run docker exec billing-web ping -M do -s 1422 billing-ledger and expect success, then -s 1423 and expect failure. Send a large POST through the frontend and confirm it does not hang. Finally, attempt a connection to UDP 514 from outside the swarm security group and confirm it is refused, then confirm the intended internal path still works.