CloudGuard Behind a Gateway Load Balancer
Fabrikam runs an AWS landing zone with a shared security VPC 10.0.0.0/16 and a CloudGuard Network Security cluster deployed as an Auto Scaling group behind a Gateway Load Balancer, with two appliances today and a maximum of six. Two spoke VPCs, 10.1.0.0/16 (applications) and 10.2.0.0/16 (data), attach through a transit gateway; each spoke route table sends 0.0.0.0/0 and the other spoke's CIDR to the GWLB endpoint. After scale-out was enabled, three problems appeared. TLS connections to a SaaS provider and to the data tier time out roughly 3-5% of the time and succeed on retry. GWLB target health flaps exactly when a new appliance registers. Appliance CPU never exceeds 45%. Separately, the data team's DynamoDB calls started failing after cutover, and compliance requires that application-to-database traffic be inspected while bulk S3 traffic must not be.
1. Given that CPU is not saturated, what are the most probable causes of the 3-5% TLS timeouts and the health-check flapping, and how would you prove each one?
Read the worked answer
1. MTU / fragmentation (most likely for selective TLS failures). GWLB uses jumbo frames (8500 bytes) toward the appliance while the spoke and internet paths are 1500. Handshakes and small packets pass; full-size data packets or large certificate chains get dropped, so the session hangs or resets and a retry often succeeds. Prove it by capturing on the appliance with tcpdump during a failure and looking for retransmissions and missing ICMP "fragmentation needed" messages, and by testing a large ping with the DF bit set across the path. Fix by aligning interface MTU/MSS on the CloudGuard side and clamping MSS toward the spokes.
2. Flow asymmetry. GWLB only keeps a flow symmetric if both directions return through the same endpoint. Check the spoke route tables for a more specific route that sends the reply path around the appliance, and check whether the return traffic from the data VPC uses a different attachment.
3. Health-check flapping on scale-out. A newly registered instance fails health checks because it is not yet fully operational: no policy installed, management connectivity not established, or the ASG lifecycle hook completes before the gateway is ready. Prove it by correlating the flapping timestamps with instance launch events and the gateway's own install log, and by testing the health-check port manually against the new instance.
2. How would you implement the requirement that application-to-database traffic is inspected while bulk S3 traffic is not, and what would you check first about the DynamoDB failures?
Read the worked answer
Inspection design. The GWLB inspects whatever is routed to it, so the lever is routing plus policy, not policy alone.
- Keep the apps spoke route for 10.2.0.0/16 pointing at the GWLB endpoint, so application-to-database traffic is inspected. Enforce on the gateway with an explicit rule and, if needed, a dedicated Threat Prevention profile for that segment.
- For S3, add a route in each spoke route table for the AWS-managed S3 prefix list pointing directly at the S3 gateway VPC endpoint. A prefix-list route is more specific than 0.0.0.0/0, so S3 traffic never reaches the appliance and bulk transfer cost and latency stay off the inspection path.
- Document that this is a routing bypass, not a policy exemption, so an auditor can see why the traffic is not inspected.
DynamoDB failures – check first. Gateway endpoints (S3 and DynamoDB) only work when the prefix-list route exists in the route table of the subnet where the traffic originates and the traffic is not NATed or forced through an appliance. After cutover the default route points at the GWLB endpoint, and the DynamoDB prefix-list route was very likely removed or overridden. Verify the spoke route tables for the DynamoDB prefix list, confirm no NAT is applied to those flows, and check the endpoint policy and the security groups on the endpoint ENIs.
3. The business now asks for a scale-out design that will not break established sessions. What would you change, and what residual risk would you accept in writing?
Read the worked answer
Changes.
- Slow the scaling trigger down. Scale on sustained throughput plus connection count over several evaluation periods, not on a single spike; the cluster already runs at 45% CPU, so headroom exists.
- Warm the new instance before it takes traffic. Use a lifecycle hook that verifies SIC to the management server, confirms the policy is installed, and passes the GWLB health check before the instance is registered as healthy.
- Guarantee flow symmetry. Register appliances per Availability Zone so both directions of a flow use the same appliance set, and make sure no route can return traffic around the GWLB.
- Fix the MTU before scaling. Scaling multiplies the number of paths affected by the jumbo-frame mismatch.
- Pin long-lived sessions. Keep a scale-in policy that drains connections (deregistration delay) instead of terminating instances with active flows.
Residual risk to accept in writing. With a stateful appliance pool, a flow that lands on an instance that is removed or fails will be reset; there is no state sharing between appliances. That means a small number of connection resets during a scale-in event or an appliance failure, mitigated but not eliminated by drain delays. Document the expected blast radius, the retry behaviour of the applications, and the fact that compliance-critical inspection depends on routing being correct after every change.