PCI segmentation where 30% of east-west traffic never reaches the ToR
A payments provider runs 600 VMs on 40 ESXi hosts across two VSX-attached racks. All VMs sit in one /16 VLAN. The 120 PCI-scoped VMs must be isolated from everything except a payment gateway and a logging collector. Security wants east-west policy enforced at the ToR, no firewall hairpin for ordinary traffic, and a rule set a network engineer can review in a pull request. Two facts complicate this: roughly 30% of VM-to-VM traffic never leaves a single ESXi host, and VM IP addresses are static and assigned by the application team, so there is no 802.1X or DHCP fingerprint to key policy on. The team owns ClearPass but has never used it for data-centre workloads.
1. Where should enforcement actually sit, and which mechanism would you choose given these constraints?
Read the worked answer
First, close the visibility gap - this dominates every product decision. A ToR ACL is blind to intra-host traffic. If a PCI VM can share an ESXi host with a non-PCI VM, the leaf is not an enforcement point for that traffic, full stop. Either apply the hypervisor's distributed firewall for intra-host rules, or use anti-affinity/placement so PCI VMs live on dedicated hosts and all their traffic is physical. Decide that first; the leaf policy is only complete once you have.
Second, choose by state requirements, not by vendor feature list. AOS-CX ACLs are stateless: you write both directions and manage ephemeral ports. For a documented application matrix (PCI to gateway, PCI to collector) that is workable, line-rate, and hairpin-free. Anything needing session state - PCI egress to the internet, session logging for audit - goes through a service leaf with a firewall, and only for the PCI scope.
Third, identity - do not over-promise. With static IPs and no 802.1X, ClearPass role download will not key on a VM identity. Realistic options: IP-based classes per tier, kept in source control and reviewed like code; or port-group-per-tier so the grouping follows inventory rather than addresses. If your AOS-CX release supports group-based policy in the EVPN overlay, that scales far better than per-IP ACLs - but verify the feature set on your exact version and hardware before writing it into the design.
2. How do you keep policy consistent across the VSX pair and roll it out without a self-inflicted outage?
Read the worked answer
Consistency: with anycast/active-gateway, traffic can ingress either VSX peer, so the ACL and class definitions must be byte-identical on both. Generate them from the same template in the same pipeline run; never hand-edit one peer. Verify with show vsx status (synchronisation healthy) and compare show access-list hitcounts per peer - an asymmetric hit profile usually means the policies differ or that one peer is carrying all the traffic.
Rollout discipline:
- Push in permit-with-log mode first, run it for a full business cycle (include month-end batch jobs, not just a Tuesday afternoon), and build the hit matrix from the logs.
- Turn that matrix into a reviewed rule set in Git - the pull request is the audit artefact.
- Take a named checkpoint on every device before the deny flip, and test the restore path in the lab before you need it.
- Apply to one rack, then the other; never both in the same window.
Common failure to plan for: deny rules that break monitoring, backup, or the hypervisor management network, which nobody listed in the application matrix. Add a permit for the management and storage networks explicitly, with a comment, rather than discovering them at 03:00.
3. How do you meet the PCI isolation requirement without hairpinning all east-west traffic through a firewall?
Read the worked answer
Segment, then inspect selectively. Put the 120 PCI VMs in their own VRF/VLAN with a default-deny at the leaf. Only flows that genuinely need state - PCI egress and the payment gateway leg - are redirected (policy-based routing or a service VRF) to a firewall on a service leaf. Everything else, including all non-PCI traffic, stays distributed at the leaf with no hairpin.
What this buys you: the firewall inspects a small, predictable flow set instead of the whole fabric, latency for ordinary traffic is unchanged, and the PCI scope shrinks to something an auditor can walk through.
What it costs you: two policy models to maintain (stateless at the leaf, stateful at the service leaf) and the requirement that the firewall sees both directions - which means the PCI VRF must be routed symmetrically through the service leaf, not leaked around it.
Verify: a path trace from a PCI VM to the gateway shows the service leaf; the same trace from a PCI VM to the collector shows the leaf ACL only; a trace from a non-PCI VM to another non-PCI VM on a different host never leaves the leaf pair. Confirm on the firewall session table that PCI egress shows both directions and that no non-PCI flow appears at all.