Hybrid Throughput Collapse Between NC2 and On-Premises NFS
Meridian Health runs an NC2 cluster in a 10.50.0.0/16 VPC and connects to on-premises 172.16.0.0/16 through a Transit Gateway and a 1.25 Gbps Site-to-Site VPN with two tunnels. Nightly backups from 40 NC2 VMs to an on-prem NFS server at 172.16.30.20 average only 180 Mbps and stall every few minutes. A ping with a 1,472-byte payload succeeds; 1,473 bytes fails. The core firewall blocks ICMP. The NC2 VPC has a NAT gateway, an S3 gateway endpoint, and a route table that sends 0.0.0.0/0 to the NAT gateway; 172.16.0.0/16 goes to the TGW. S3 backup traffic is crossing the NAT gateway and the team wants to stop that. One VPN tunnel flaps daily. The CISO asks whether to fund Direct Connect and what to fix first.
1. Diagnose the NFS throughput problem and recommend specific MTU, MSS, and path changes.
Read the worked answer
The 1,472/1,473 ping result is the classic PMTUD black-hole signature. The AWS VPN path MTU is 1500 bytes, while large NFS writes set the Don't Fragment bit and depend on ICMP Fragmentation Needed to learn the lower MTU. The core firewall drops that ICMP, so TCP retransmits and stalls; the link is not actually saturated at 180 Mbps. Fix the path before buying more bandwidth: permit ICMP type 3 code 4 on the core, clamp TCP MSS on the NFS clients to roughly 1360–1400 bytes, and set the client interface or route MTU to 1400 for traffic over the VPN. Verify with tracepath and large sequential writes, not just ping. Also check whether the NC2 overlay or CVM interfaces are advertising jumbo frames toward the VPN; jumbo must not be used across the VPN. Stabilize the flapping tunnel with BGP preferences and both tunnels active so the fix is not masking a failover problem.
2. How should the team correct the S3 backup path so traffic stops using the NAT gateway?
Read the worked answer
Attach the S3 gateway endpoint to the correct NC2 subnet route table and confirm that the endpoint prefix list route is present and more specific than the default route. A gateway endpoint does not use ENIs or the NAT gateway; it is a route-table target, so missing or misassociated route tables are the usual cause. Check that the endpoint policy allows the bucket and that the instance security group permits HTTPS to the S3 prefix list if the group is egress-restricted. Then verify DNS and proxy behavior: if the backup tool uses an HTTP proxy or resolves S3 to a public IP and routes it through the default route, it will still traverse NAT. Test from a VM with a direct S3 upload and confirm the NAT gateway byte count does not increase. Keep the NAT gateway for other egress, but do not use it as a substitute for a missing endpoint route.
3. Should the CISO fund Direct Connect now? Justify the decision with evidence.
Read the worked answer
Not yet. The observed 180 Mbps is well below the 1.25 Gbps VPN capacity, and the ping result points to an MTU black hole, not a bandwidth ceiling. Direct Connect would not automatically fix an ICMP-blocking firewall or a bad MSS setting, so the team could spend money and still see stalls. First fix PMTUD, stabilize both VPN tunnels, and measure sustained throughput over a multi-hour backup window. If the corrected VPN sustains a predictable rate that meets the backup SLA, keep it. If the organization needs more deterministic latency, higher sustained throughput, or a private path that avoids internet-grade variability, then evaluate Direct Connect or a Direct Connect gateway with a transit VIF. The decision should be based on post-fix throughput, latency sensitivity, and cost, not on the current symptom.