PCI Segmentation in a Shared Data Center With a Cloud On-Ramp
Vantage Payments runs a three-pod spine-leaf data center (32 leaves, 40/100G) shared by PCI cardholder-data workloads (300 VMs), corporate IT, and a new container platform. A 10 Gbps dedicated connection to AWS us-east-1 carries 4 TB per day of batch plus an API tier that must fail over within 60 seconds. Audit requires cardholder traffic to be segmented from all other traffic, with enforcement points that log every denied flow, managed by a team of three that does not control the hypervisor. Segmentation is currently enforced by a pair of 10 Gbps firewalls on a services leaf; measured east-west PCI traffic peaks at 22 Gbps during the 23:00-01:00 settlement window, and the firewalls run at 91% CPU. Cloud egress traverses the same firewall pair. The firewall refresh cycle is five years and the company is in year four.
1. Where should PCI segmentation be enforced, given the volume, the audit wording, and the team size?
Read the worked answer
Work the requirement backwards. The audit needs segmentation plus logged denials, not stateful inspection of every packet. Measure the traffic before choosing a device: 22 Gbps of east-west peak against a 10 Gbps firewall pair already at 91% CPU means the current design is not just undersized, it is almost certainly adding latency and dropping during settlement.
Options and why they lose or win:
- Security groups in the hypervisor: excluded by the requirement and by team ownership.
- Stateful firewalls for all east-west: would need dual 40 Gbps devices with logging, which is a refresh brought forward two years plus a capacity model that grows with the container platform.
- Leaf ACLs: line rate at 40/100G, but no stateful logging; denials are visible only through ACL counters or NetFlow, which satisfies the letter of the requirement if audit accepts flow logs.
Recommend a two-tier split. Use leaf ACLs for the high-volume, well-understood PCI-to-PCI and application-to-database flows - that is the bulk of the 22 Gbps and it is policy, not inspection. Keep the firewall pair for north-south and for PCI-to-corporate crossings, which are a small fraction of the volume, so the devices stay well under capacity and their logs carry the audit narrative.
If audit insists on stateful inspection of all east-west PCI flows, the answer is a funded 2x40 Gbps refresh this year. Say that plainly rather than pretending the existing pair can absorb 22 Gbps.
2. Design the AWS connectivity to meet the 60-second failover requirement.
Read the worked answer
A single 10 Gbps Direct Connect is a single failure domain, and 60 seconds is not achievable over static routing or default BGP timers. A DX circuit rebuild after a location failure takes weeks to months, so resiliency has to be provisioned, not improvised.
Recommend dual Direct Connect circuits terminating at two different Direct Connect locations, with MACsec where available, BGP with BFD for sub-second detection, and an Internet VPN as a third path kept permanently established. Advertise the VPN with a lower local preference so it is pre-installed in the FIB and failover is a forwarding change, not a session negotiation. Default BGP hold time of 180 seconds alone breaks the 60-second target before any application consideration.
Scope by traffic class rather than treating all cloud traffic the same. The 4 TB per day batch can tolerate a VPN fallback and even a slower window. The API tier cannot tolerate an unplanned rebuild, so it gets the dual-DX plus VPN treatment and health-checked DNS or a load balancer with TTLs under 30 seconds.
Test the failure, not the diagram: shut one DX location in a maintenance window and measure end-to-end API error rate and time to recovery, including DNS and application health-check timing.
3. What common-mode and failure-domain risks would you flag before this design is approved?
Read the worked answer
The firewall pair is the most dangerous common mode. It sits in the PCI enforcement path and in the cloud egress path, is at 91% CPU, and is a year from refresh. A single device failure or a policy push takes out both settlement traffic and cloud API connectivity at once. Splitting the roles - dedicated enforcement for the corporate boundary and a separate, sized path for cloud egress - removes the coupling.
Other risks to put in front of the approver:
- Reflector and control-plane concentration: if all BGP sessions to AWS land on one pair of border devices, dual DX buys nothing.
- DNS and load balancer TTLs: a 300-second TTL makes 60-second failover impossible regardless of routing.
- The container platform: 300 PCI VMs today may become thousands of ephemeral endpoints; leaf ACL and policy models must be able to reference identity or tags, not IP addresses, or the design will not survive 18 months.
- Logging volume and retention: 22 Gbps of inspected flows generates log volumes the three-person team cannot triage; decide what is sampled and what is retained before audit asks.
Close with an explicit dependency list: capacity sizing, refresh timing, DNS ownership, and audit sign-off on flow-based logging. Any of these unowned turns into a project risk later.