Inspection VPC Bypass and NAT Port Exhaustion in AWS

The TGW is doing exactly what it is told. Each attachment is associated with one route table and propagates its routes into whichever tables are configured. When the analytics VPC was attached and began propagating its CIDR into the table that spoke VPCs use, spokes gained a direct path to the analytics range and no longer matched the default route pointing at the inspection VPC. Traffic still flows, which is why nobody noticed until logs were checked.

The fix is two route tables with a deliberate structure:

• A spoke route table: associated with all spoke attachments, containing 0.0.0.0/0 pointing at the inspection VPC attachment. Do not enable propagation of spoke CIDRs into this table, so every spoke-to-spoke flow must traverse the inspection VPC.
• An inspection route table: associated with the inspection VPC attachment, containing the spoke CIDRs as static or propagated routes so the FortiGate can return traffic to the correct spoke.

Symmetry needs two more settings. Enable appliance mode on the inspection VPC attachment so the TGW pins a given flow to one FortiGate in one AZ for its lifetime; without it, return packets can take a different AZ and hit a firewall that never saw the flow, producing silent drops. Confirm source/destination check is disabled on the FortiGate ENIs, otherwise AWS discards traffic not addressed to the instance.

Verify with TGW route-table lookups from the analytics subnet and with FortiGate session and forward-traffic logs before declaring it fixed. Then add a change-control step that reviews route-table propagation for every new VPC attachment.

The symptom set points to source NAT port exhaustion, not bandwidth. Roughly 64,000 source ports are available per public address per destination tuple, so a single elastic IP and 60,000 sessions toward one API endpoint is by definition near the ceiling, and new connections fail once ports cannot be allocated. Long-lived connections from many users to one destination make this worse because ports are held open.

Options, in order of effort:

• Add public addresses. Attach multiple secondary private IPs with associated elastic IPs to the FortiGate ENI, then configure an IP pool spanning them, with the outbound policy using that pool. This multiplies the port space and spreads load. Watch AWS limits per ENI and per account, and confirm the pool addresses are in the same subnet as the ENI.
• Reduce consumption. Shorten idle timeouts for that destination so dead sessions release ports, and check for connection leaks or retry storms in the application.
• Split the traffic. Send that SaaS destination, or a category of trusted SaaS, out through an AWS NAT gateway and keep the FortiGate for inspected traffic. This removes the pressure but also removes inspection for that destination, so it must be a conscious risk decision documented with the security owner.
• Redesign at scale. If growth continues, plan a load-balanced, autoscaling egress architecture rather than repeatedly adding addresses to a fixed pair.

Short term, add the IP pool during a window and monitor NAT allocation failures and session counts per destination. Confirm the SaaS provider does not whitelist the old single address before you switch to a pool, or the fix becomes an outage.

The 90 seconds is dominated by elastic IP reassociation and route and ARP convergence in the AWS network, not by FortiGate heartbeat detection. FGCP over unicast in AWS already needs non-standard settings because multicast is unavailable, and the elastic IP mobility mechanism works, but its completion time is not something you can tune down to single-digit seconds reliably. Any design that depends on one instance failing and its public address moving will inherit that floor.

Realistic options:

• Accept and design around it: keep FGCP, reduce blast radius by shortening TCP keepalives and session timers so dead sessions are reaped quickly, and make clients retry. Measure the real failover time with injection tests rather than assuming. This is cheap but does not meet a strict SLA.
• Decouple the address from the instance: front the FortiGates with a Gateway Load Balancer and target group in both AZs. New flows are hashed to a healthy target, and a failed target is removed without waiting for an address to move. Existing flows still break, but recovery for new connections is fast. This changes the architecture and the inspection insertion point, so it needs a lab validation.
• Dual-path routing: attach both FortiGates to the TGW and advertise the same prefixes with a routing preference, so failure is handled by route withdrawal rather than address movement. This is more moving parts and needs careful asymmetry testing with appliance mode.

Recommendation: validate the Gateway Load Balancer option in a non-production VPC first, because it addresses the actual constraint, and keep FGCP as the fallback while measuring whether the business can tolerate the current behaviour in the meantime.