Transit Gateway Asymmetric Return Path with Dual Attachments
A company runs a Transit Gateway with two VPC attachments in the same Region. VPC-A hosts application subnets 10.10.0.0/24 and 10.10.1.0/24. VPC-B hosts a shared-services VPC with a stateful inspection appliance in an Auto Scaling group behind an internal Network Load Balancer. Each VPC has its own attachment subnet in a different Availability Zone. The appliance inspects traffic between VPC-A and an on-premises network reached over a Site-to-Site VPN attached to the same Transit Gateway. After a scaling event replaced one appliance instance, roughly half of the connections from VPC-A to on-premises time out, while the other half succeed. VPC flow logs show SYN packets reaching on-premises but no return traffic for the failing flows. Transit Gateway route tables for both attachments contain identical propagated routes.
1. Explain why only roughly half of the flows fail and describe the specific Transit Gateway and appliance configuration changes that make return traffic symmetric.
Read the worked answer
The failure is an asymmetric return path caused by the appliance being stateful while the Transit Gateway makes independent forwarding decisions per packet, not per flow.
Why half fail
- VPC-A sends traffic to the appliance attachment subnet. The Transit Gateway chooses among the appliance attachment's subnets (one per AZ) based on the destination IP and its own hashing, not on which appliance instance the flow was steered to.
- The surviving appliance instance holds state for the flow, but the return packet from on-premises arrives at the Transit Gateway and is forwarded to the other AZ's attachment subnet, or directly back to VPC-A, bypassing the appliance that holds the state.
- Because the appliance is stateful, a packet arriving on an interface with no matching state is dropped. Roughly half the flows land on the instance that never saw the SYN, so their return traffic is discarded and the connection times out.
Fixes
- Make the appliance attachment appliance mode enabled. Appliance mode keeps the forward and return path of a flow pinned to the same attachment subnet/AZ, so the same appliance instance sees both directions.
- Ensure the appliance is deployed one instance per AZ and that each attachment subnet exists in every AZ used by VPC-A and the VPN. A single appliance in one AZ cannot satisfy appliance mode for flows originating in the other AZ.
- Verify that the appliance's security group and the VPC route tables send all traffic (including return traffic to on-premises) back through the Transit Gateway attachment, never directly to the VPN or internet gateway.
- Confirm the Transit Gateway route tables for the VPC-A and VPN attachments both point to the appliance attachment for the relevant prefixes, and that no more-specific static route bypasses it.
- After enabling appliance mode, re-test with new flows; existing flows keep their old path until they age out.
Appliance mode is the decisive control here: without it, the Transit Gateway has no mechanism to keep a flow's two directions on the same attachment subnet, and any stateful middlebox will drop half the return traffic.
2. A teammate proposes removing the appliance entirely and letting the Transit Gateway route VPC-A traffic directly to the VPN attachment. What operational and security consequences must be evaluated before accepting this?
Read the worked answer
Removing the appliance is a valid simplification, but it changes the security and observability posture in ways that must be assessed explicitly.
Security consequences
- All inspection, egress filtering, intrusion detection, and TLS break-and-inspect currently performed by the appliance disappears. Traffic between VPC-A and on-premises becomes Transit Gateway-to-VPN with only security groups, NACLs, and any on-premises firewall in the path.
- Segmentation between VPC-A and shared services relies entirely on Transit Gateway route table separation and security group references. If the route tables currently send all inter-VPC traffic through the appliance, direct routing widens the blast radius unless route table associations and propagations are tightened.
- Compliance controls that require a centralized inspection point (for example, a documented network boundary with logged inspection) may no longer be met, so the governance owner must sign off.
Operational consequences
- You lose flow-level visibility from the appliance; you must compensate with VPC Flow Logs on the attachment subnets, Transit Gateway Flow Logs, and VPN tunnel metrics.
- Scaling and AZ-failure behavior improves because there is no stateful middlebox to keep symmetric, but you also lose the ability to apply layer-7 policy.
- The VPN's available bandwidth and its two tunnels become the direct path for VPC-A traffic; verify the tunnel capacity and that BGP advertisements still select the intended path.
- Return traffic symmetry is no longer a concern, which removes the appliance-mode dependency and a recurring class of outages.
Decision guidance
If inspection is a hard requirement, keep the appliance but fix it with appliance mode and per-AZ deployment. If inspection is not required and route table isolation plus security groups are sufficient, direct routing is simpler and more resilient, but document the accepted risk and update the compliance evidence.