Centralized Inspection with Transit Gateway and Gateway Load Balancer
RetailCo has three AWS accounts: prod, dev, and shared-services. Each account has two VPCs in us-east-1 across two availability zones. Peak inter-VPC traffic is 500 Mbps; internet ingress is 200 Mbps. Security requires all east-west and ingress traffic to pass through CloudGuard Network Security gateways, but the design must survive one AZ failure and scale to 1 Gbps without manual intervention. The current design puts one gateway instance in each VPC, managed by SmartConsole; policy is inconsistent and the team cannot see traffic between VPCs. The network lead proposes a security VPC with a Transit Gateway and Gateway Load Balancer. A colleague warns that Transit Gateway will break return traffic and cause asymmetric drops. The budget allows two gateways per AZ but no third-party SD-WAN, and no new inspection product.
1. Propose where to place the CloudGuard gateways and how traffic should be routed for east-west and ingress flows.
Read the worked answer
Place the gateways in a dedicated security VPC in the shared-services account, across two AZs, behind a Gateway Load Balancer. Use a Gateway Load Balancer endpoint in every spoke VPC and in an ingress VPC. Do not put a gateway in each spoke VPC; that recreates the policy-drift and visibility problem.
Routing:
- Spoke east-west: spoke route tables send other VPC CIDRs to the Transit Gateway. The TGW route table sends those prefixes to the security VPC attachment. The security VPC attachment must have appliance mode enabled so a flow stays on the same gateway ENI and AZ in both directions.
- Spoke to internet: spoke route tables send 0.0.0.0/0 to the local GWLB endpoint. The GWLB forwards to a healthy CloudGuard gateway. The gateway applies policy and sends traffic to the ingress VPC or NAT egress path according to design.
- Ingress: terminate the internet in an ingress VPC with an internet gateway, route inbound traffic to a GWLB endpoint, inspect, then forward over TGW to the destination spoke. Return traffic follows the same pinned path.
- Scale and failure: use an autoscaling group per AZ with CloudWatch alarms on GWLB metrics. Two gateways per AZ gives N+1 within an AZ and survives one AZ loss at the required 1 Gbps.
The key is that the GWLB and TGW appliance mode provide flow stickiness; the gateway itself is not asked to solve asymmetry with routing tricks.
2. Explain what Transit Gateway appliance mode does and why the colleague's warning is partly right but not a reason to abandon the design.
Read the worked answer
Transit Gateway appliance mode is an attachment setting that keeps a flow on the same appliance ENI and AZ for the life of the flow. Without it, TGW can choose a different path for return packets, and a stateful gateway that did not see the first packet will drop the return traffic. That is the real failure mode behind 'asymmetric drops'.
The colleague is right that naively attaching a stateful appliance to TGW can break return traffic. The warning is incomplete because appliance mode is the supported control for that exact problem. Enable it on the security VPC attachment, verify that the GWLB target groups are zonal and that health checks are passing in both AZs, and confirm that the gateway cluster state sync is working. Also check that no spoke route sends the same prefix to two different next hops with different metrics that could flap mid-flow.
Abandoning the design would leave per-VPC gateways with inconsistent policy and no east-west visibility. The correct response is to treat appliance mode, GWLB flow stickiness, and cluster state sync as mandatory design elements, not optional tuning.
3. Why did the original one-gateway-per-VPC design fail operationally even though each VPC had inspection?
Read the worked answer
Each VPC had a gateway, but the gateways were managed as separate policy targets with no shared object model or consistent rulebase. That produced three operational failures.
First, policy drift: a rule added in prod was not added in dev or shared-services, so the same application flow was allowed in one VPC and blocked in another. Second, no east-west visibility: traffic between VPCs left one VPC and entered another, but there was no central point that saw the full flow, so the team could not distinguish a blocked flow from a routing problem. Third, scaling was per VPC and manual: each gateway had to be sized and patched separately, and an AZ failure in one VPC required a manual rebuild.
The centralized security VPC does not automatically fix policy; it makes policy consistent because there is one enforcement point and one set of gateway targets. It also centralizes logging and lets the team scale inspection capacity independently of the number of VPCs. The trade-off is that the security VPC and its attachments become a shared dependency, so health checks, AZ spread, and change control must be stronger than they were in the per-VPC model.