Centralized Inspection VPC: Asymmetric Resets and Appliance Saturation
An enterprise runs three AWS VPCs (prod, shared services, inspection) attached to a Transit Gateway. On-premises connects over a 10 Gbps Direct Connect with a 1 Gbps IPsec VPN as backup. Security policy requires all east-west and egress traffic to be inspected by FTDv appliances in the inspection VPC, published behind a Gateway Load Balancer. After cutover, application teams report intermittent TCP resets on long-lived connections between prod and shared services, and FTDv CPU sits at 90% while aggregate throughput is only 2.1 Gbps. The GWLB target group reports both targets healthy, but VPC flow logs show the two directions of the same connection arriving at different FTDv instances. Budget allows two additional FTDv appliances.
1. Diagnose the intermittent resets and identify the two most probable root causes.
Read the worked answer
Mechanism: the FTDv is a stateful device. A flow only survives if both directions traverse the same appliance instance. GWLB hashes on the packet 5-tuple, so symmetry depends on nothing in the path changing that tuple or bypassing the LB.
Root cause 1 - path asymmetry. VPC flow logs already show the two directions landing on different instances. The usual culprits: (a) the FTDv performs SNAT from a per-instance pool, so the return packet's 5-tuple hashes to a different target; (b) a TGW route table or subnet route sends return traffic directly to the TGW instead of back through the GWLB endpoint; (c) cross-zone load balancing combined with an appliance that rewrites the destination. Confirm by adding the flow's source/destination pair to the hash comparison and by checking whether the resets correlate with NAT sessions.
Root cause 2 - target health flap or scale event rehashing existing flows. When a target is added/removed or deregisters, GWLB rehashes new flows; long-lived connections that fail to rehash mid-stream break. Look for target group deregistration events or a failed health check around the reset timestamps.
Fix direction: keep stateful inspection in the return path (do not bypass the GWLB endpoint), avoid SNAT on the FTDv, and enable connection draining / longer deregistration delay so flows are not cut mid-stream.
2. Given 2.1 Gbps at 90% CPU with two FTDv instances and budget for two more, how would you scale, and what would you check first?
Read the worked answer
Check before buying: 90% CPU at 2.1 Gbps is suspicious. Verify (1) which FTDv performance tier / vCPU-vRAM entitlement is actually licensed - throughput is capped by the license, and running a small tier with heavy inspection produces exactly this profile; (2) whether Snort 3 with a full rule set plus TLS decryption is enabled on all traffic, since decryption can cut throughput several-fold; (3) whether one instance is carrying most flows (see the flow-log asymmetry).
Scaling decision:
- Horizontal is correct here, because the failure is per-flow CPU on inspection, and GWLB is designed to spread flows. Two additional instances, matched to the licensed tier, take the pool to four.
- Do not scale vertically alone - a single larger instance still cannot exceed the tier's throughput and re-creates the same headroom problem at the next growth step.
- Prerequisite for horizontal scaling is session symmetry. Adding instances before fixing the asymmetry doubles the number of broken flows. Fix the hash/NAT/return-path issue first, then add capacity.
- Sizing math: if 2 instances deliver 2.1 Gbps at 90% CPU, useful capacity is roughly 2.3 Gbps; four instances give roughly 4.6 Gbps, minus headroom. Plan for 60-70% steady-state utilization to absorb failover of one instance.
- Consider flow offload and removing decryption from flows that policy does not require, before assuming more appliances are the answer.
3. Which traffic should you argue should NOT be steered through the inspection VPC, and how would you justify it?
Read the worked answer
Candidates to exclude, with justification:
- TGW-to-on-premises backup VPN and Direct Connect health probes / BGP. These are control-plane and must not depend on a stateful appliance; inspecting them means an appliance failure breaks routing, not just data.
- AWS service endpoints (S3 gateway endpoints, ECR, CloudWatch Logs). Traffic to a gateway endpoint never leaves the VPC and is already authenticated by IAM and encrypted; forcing it through a GWLB adds latency and appliance load for traffic you cannot meaningfully inspect (TLS to AWS endpoints).
- Backup/replication and database sync between prod and shared services if already mutually authenticated and encrypted - but only after confirming with the security owner that the control objective (data exfiltration detection) is met by another mechanism such as VPC flow logs plus DNS logging.
- Instance metadata and internal health-check traffic.
How to justify: frame it as a control-coverage argument, not a performance argument. For each exclusion, state the compensating control (IAM policy, security group reference, gateway endpoint policy, flow/DNS logging) and the residual risk. Anything carrying regulated data or crossing a trust boundary - egress to the internet, any prod-to-third-party flow - stays in scope. Present the exclusions as a documented exception list with an owner and review date, and quantify the offload so the security team sees the capacity it buys.