MTU 9000 database, MTU 1500 API tier, and an ICMP block
Your team runs a three-tier application in a single datacentre. Web servers on 10.20.10.0/24 call api servers on 10.20.11.0/24, which open PostgreSQL connections to db-prod-01 at 10.20.20.15:5432. The database and storage NICs were built with MTU 9000 two years ago; the api servers were added later and are still at MTU 1500. On Tuesday the network team tightened a firewall rule so that only TCP 5432 is permitted between 10.20.11.0/24 and 10.20.20.0/24, blocking all ICMP. Since then, report exports from the web UI, which return 50 to 500 MB, hang for 30 to 90 seconds and fail with 'connection reset by peer', while the dashboard and small queries are unaffected. On api-prod-04, ss -tan shows a few connections in SYN-SENT, and ping -M do -s 8972 10.20.20.15 fails immediately with 'Message too long'. A capture on the api host shows 1500-byte segments retransmitted about six times before the reset. The firewall team can make a change within 24 hours but needs a precise rule, and the application owner wants a fix that survives the next firewall review.
1. Identify the failure mechanism that explains the evidence. Separate the misleading observations from the diagnostic ones, and state exactly what you would capture to confirm the diagnosis.
Read the worked answer
The mechanism is a PMTUD blackhole caused by blocking ICMP type 3 code 4.
Walk the path in both directions. API segments are at most 1500 bytes including headers, so they traverse the 1500-byte path to the database without trouble: the SYN, the query, and small result sets all succeed. The database NIC is at MTU 9000, so its responses can be up to 9000-byte IP packets with the DF bit set. The router that forwards toward 10.20.11.0/24, which is a 1500-byte segment of the fabric, cannot fragment a DF packet. It is required to drop the packet and return ICMP type 3 code 4 (fragmentation needed, DF set). Tuesday's rule change now discards that ICMP, so the api host never learns to lower its path MTU estimate, the database retransmits full-size responses that keep being dropped, and after the application's timeout the connection is reset. This matches exactly: small queries fine, large result sets fail after a long hang.
Two observations are distractors. ping -M do -s 8972 failing immediately with 'Message too long' is the local stack refusing to emit a packet larger than the api NIC's own 1500 MTU - it tests nothing about the path. A handful of SYN-SENT entries is worth a second look but does not explain a 30-90 second stall followed by a reset on established connections; check whether they persist or are a conntrack artefact.
Confirm by capturing ICMP on the api host and the database host during a failing export, reading the firewall deny counters for ICMP between the two subnets, running tracepath -n 10.20.20.15 from the api host to watch the PMTU estimate collapse and stall, and using ss -ti to show retransmissions and a collapsing congestion window on the affected connection.
2. Give the candidate fixes, then choose one given that the firewall owner needs a precise rule. Justify the choice, including why raising the api MTU alone is not sufficient.
Read the worked answer
Options.
- Permit ICMP type 3 code 4 (and type 11, time exceeded) between 10.20.11.0/24 and 10.20.20.0/24 in both directions. RFC 4890 treats frag-needed as required for correct operation, not optional.
- Lower the database NIC to MTU 1500 so it never emits oversized packets toward the api tier. Fast, one host, but throws away jumbo frames on a path where the storage network may genuinely be 9000 end to end.
- Clamp MSS on the router for TCP/5432:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu. No application change, no ICMP dependency for TCP. - Raise the api servers to 9000 and verify every hop.
Recommendation. Option 1 is the correct fix and the one that keeps working for every protocol, and it is expressible as a precise rule: permit ICMP type 3 code 4 from 10.20.20.0/24 to 10.20.11.0/24 and the reverse, and log it. Pair it with option 3 as an immediate mitigation if the firewall change slips, because the export failure is customer-visible now.
Why not just raise the api MTU. It may fix this path today, but any future hop that is not 9000 - a VPN, a load balancer, a new subnet - reintroduces the same failure with no ICMP to recover, and it silently doubles memory and buffer pressure on hosts sized for 1500. MTU must be consistent and verified end to end, or PMTUD must work. Reaching 9000 on the api tier is a separate, tested change.
State in the change record that option 3 hides a network defect rather than fixing it, and that option 1 must still land.
3. How would you validate the fix off-peak, and what monitoring or change-process control prevents this class of outage from recurring?
Read the worked answer
Validation. Before the change, record a baseline from api-prod-04: run the failing export against a non-production replica or a read-only query returning several hundred MB, and capture duration, ss -ti retransmit counts, and firewall ICMP counters. After the rule is applied, repeat the same query and confirm completion in seconds, retransmits near zero, and ICMP frag-needed now permitted at the firewall rather than denied. Run tracepath -n from one api host to the database and confirm the path MTU resolves to 1500 without stalling. Do it on one host first, then all, and keep the previous rule text ready to restore.
Monitoring gap. Every check that passed during this outage was a port check: TCP 5432 was reachable and the dashboard worked. Add a synthetic that transfers a payload larger than the smallest MTU on the path between each tier pair - a scripted query returning 20 MB, or a plain large-file transfer - and alert on duration rather than reachability. Alert on firewall deny counters for ICMP frag-needed, since any non-zero value indicates PMTUD is being broken somewhere.
Change control. The firewall change was reviewed for ports and not for what else it removed. Require that any rule set review enumerate ICMP types, include a per-host-pair path MTU baseline in the CMDB, and record MTU as a configuration item so a host built at 1500 in a 9000 fabric is a known deviation rather than a discovery. Also require a rollback line and an owner for the post-change check, since this change had neither.