East-West Inspection at 7.8 Gbps: Architecture Choice Under a PCI Deadline
A payments company runs 22 AWS VPCs across three regions, stitched together by a Transit Gateway, with east-west inspection through two FortiGate-VMs (c5.4xlarge) in a security VPC. Peak inspected throughput reached 7.8 Gbps last month against a tested ceiling of roughly 6 Gbps, and p95 latency for card authorization calls rose from 8 ms to 31 ms. VPC flow logs show that 61% of the traffic reaching the firewalls is S3 and DynamoDB endpoint traffic that never leaves the region, and a further 22% is development and test workloads. A PCI assessor now requires all egress to be inspected and logged with twelve-month retention. A new business unit must be onboarded in six weeks. The platform team is three engineers, there is no appetite for a per-VPC firewall fleet, and incremental spend is capped at 15%.
1. Choose the target architecture and justify it against the constraints, including what you would do in the six weeks before it is ready.
Read the worked answer
No single option satisfies every constraint at once, so the answer is a phased one.
Assessment:
- GWLB with an autoscaling FortiGate-VM fleet scales horizontally and keeps the full FortiOS feature set, including the inspection profiles and logging pipeline the team already knows. Cost scales with capacity, which helps the 15% cap if you first cut the traffic actually sent to it. Operational burden is nontrivial: autoscaling groups, lifecycle hooks, health checks, and GWLB endpoint association in every VPC that needs inspection.
- FortiGate CNF removes the undifferentiated infrastructure work - no instances to patch or scale - and fits a three-engineer team well. It is the strongest pick if the required inspection policies can be expressed in the supported feature scope; verify the team's specific IPS and TLS inspection requirements against it before committing.
- Native cloud firewall is cheapest operationally but may not reproduce the existing policy set and logging format, and migrating 22 VPCs of rules is its own project.
Recommendation: the traffic reduction below is mandatory regardless of choice, because paying to inspect 61% irrelevant traffic is the largest avoidable cost. After reduction, the inspected load likely falls within the current pair's capacity, which buys the time to evaluate CNF properly rather than under deadline pressure. If CNF passes the feature check, target it for the new business unit and migrate region by region; if not, GWLB autoscaling.
First six weeks: reduce traffic, onboard the new business unit into the existing inspected path with its own policy package, and run CNF proof-of-value in parallel in the smallest region.
2. How do you cut the inspected traffic without creating a PCI blind spot?
Read the worked answer
Cut by proving irrelevance, not by hoping. Each exclusion needs evidence attached.
- S3 and DynamoDB gateway endpoints. Traffic to these endpoints is regional and does not traverse the internet or a Transit Gateway if the route tables send it to a gateway endpoint. Add the endpoints to the relevant route tables so the flows bypass the security VPC entirely. Evidence required: flow logs showing the destination prefix lists and zero corresponding egress to the public internet. This is the single largest win and it is safe because gateway endpoint traffic is not internet egress.
- Development and test workloads. Move them to a separate inspection path with a lighter profile and a separate log destination, or exclude them from the PCI-scoped inspection segment entirely, provided they genuinely hold no cardholder data. Get that in writing from the data owner - 'dev has no card data' is exactly the assumption that turns into an audit finding.
- Do not touch anything in PCI scope, including egress from any VPC that stores, processes, or transmits cardholder data, and including management-plane traffic to logging and key management. The assessor's requirement is specifically about egress, so shrinking east-west inspection of non-card traffic is defensible while shrinking egress inspection is not.
- Re-baseline after the change, not before. If p95 stays at 31 ms after traffic drops, the latency is not a throughput problem and you are about to buy capacity you do not need.
Keep a written exclusion register with owner, evidence reference, and review date. Every future assessor will ask for exactly that document.
3. Latency and correctness issues remain after the traffic reduction. What do you check, and why do they occur?
Read the worked answer
Two distinct failure modes produce these symptoms, and they are often confused.
Asymmetric routing and flow symmetry. When a Transit Gateway is used with a middlebox appliance, the appliance must be associated in a way that keeps both directions of a flow on the same firewall instance. If the TGW is not configured for appliance-mode behaviour, return traffic can take a different AZ path, which either gets dropped or hairpins back through the second firewall. Symptoms: intermittent drops, latency spikes concentrated on cross-AZ flows, and firewall session counts that look low relative to the traffic volume. Check the TGW attachment configuration and whether the security VPC endpoints are in the same AZ as the workloads being inspected.
Gateway Load Balancer hashing and flow stickiness. GWLB must send both directions of a flow to the same target. Verify flow stickiness is functioning, that firewall instances are healthy from the load balancer's perspective, and that you are not sending flows to an instance that is up but saturated.
One-arm versus two-arm design. A single-interface design simplifies routing but can double the traffic on the interface and create confusing source addresses. A two-arm design is cleaner logically but requires symmetric subnet routing in every attached VPC, which in 22 VPCs is a change management problem.
Also check scale-in behaviour. If autoscaling removes a firewall instance while sessions are active, you get exactly the p95 tail you are seeing. Add connection draining and scale-in protection so instances are not terminated with live flows.
Measure per-AZ before changing anything: if latency is normal within an AZ and poor across AZs, the answer is placement, not capacity.