Double Inspection Breaks mTLS After CN-Series Rollout
An online retailer runs 42 services on Amazon EKS in us-east-1 across six namespaces, peaking at about 3 Gbps of east-west traffic, with a VM-Series pair behind a Gateway Load Balancer for north-south inspection and CN-Series deployed as a DaemonSet on every node. Security policy is split: internet-facing rules live in Strata Cloud Manager, service-to-service rules in a Panorama template pushed by a platform team. After CN-Series was enabled cluster-wide, checkout-to-payments calls began failing with TLS handshake resets, and session logs show each request inspected twice, once on the source node and once on the destination node. The payments service uses mutual TLS with SPIFFE certificates rotated hourly, and the security team wants all east-west traffic decrypted and inspected before the next PCI audit.
1. Explain why double inspection breaks these calls and propose an enforcement topology that inspects each flow exactly once.
Read the worked answer
Each inspection point is a TLS man-in-the-middle. On the first pass the firewall presents a re-signed certificate to the client and opens its own session to the server; on the second pass it does the same again to a peer that is already expecting the service's own certificate. With mTLS, both sides verify the peer certificate, so the inner leg fails verification and the handshake resets. The doubled work also explains the latency and the extra sessions.
Topology that inspects once:
- Define one enforcement boundary per direction of trust. North-south stays on the VM-Series behind the Gateway Load Balancer; do not hairpin east-west traffic through it.
- East-west stays on CN-Series, but decide on one inspection point, for example the source node, and make the destination node pass traffic without inspecting it again. This is a policy decision expressed in the CN-Series rule set by namespace and direction, not a routing trick.
- Ensure the cluster's routing keeps a flow symmetric so the same CN-Series instance sees both directions; otherwise the inspection point is bypassed or duplicated.
Verify with session logs after the change: one decryption entry per request, matching source and destination counters, and no reset counters on the payment path. Document the boundary explicitly so a future namespace onboarding does not reintroduce a second pass.
2. The security team insists on decrypting all east-west traffic. How do you handle the SPIFFE mTLS workload and still satisfy the audit?
Read the worked answer
Decrypting hourly-rotated SPIFFE mTLS is technically possible and operationally hostile: it requires distributing a CA the services trust, re-signing certificates that rotate hourly, and accepting that any rotation failure becomes a payment outage. It also weakens the very property the audit is meant to demonstrate, because the control becomes dependent on one appliance's certificate chain.
A defensible position for the auditor:
- Exclude the payments namespace from decryption via a narrowly scoped no-decrypt rule keyed to the namespace addresses and the service port, not to a zone or a broad category.
- Show that identity-based segmentation is enforced without payload visibility: CN-Series policy that permits only specific workload identities to reach payments on 443, default-deny for everything else, and network policy in the cluster as a second layer.
- Show that the excluded traffic is still subject to connection-level controls, logging of allow and deny decisions, and anomaly detection, and that the exclusion is inventoried with an owner and a review date.
- Compensate elsewhere: decrypt the east-west flows that carry cardholder data in cleartext or use plain service-to-service TLS without mTLS, and keep full decryption on ingress and egress where data crosses the cluster boundary.
What to refuse: a cluster-wide blanket no-decrypt rule. That converts a scoped exception into an unmonitored segment and will not survive audit scrutiny.
3. Policy for these firewalls is split between SCM and a Panorama template. What governance change would you make, and why does the split matter technically?
Read the worked answer
Two management planes create two sources of truth, and the failure mode is silent: a rule edited in one place is overwritten or shadowed by the other, and neither team sees the effective policy. In a PCI scope, that is an audit finding before it is an outage.
Options, in order of preference:
- Consolidate on one management plane for both VM-Series and CN-Series, with the platform and security teams working in folders and roles inside that plane. SCM with folder scoping and role-based access gives the platform team its namespace without a second tool.
- If consolidation is impossible now, make the boundary explicit and technical: SCM owns internet-facing rules and shared objects, Panorama owns only service-to-service intent, and the interface between them is a documented address and tag convention. Add a CI check that renders the effective rule base from both and fails the pipeline on shadowing or on conflicting objects with the same name.
- Tag-based dynamic address groups keyed to Kubernetes labels and AWS tags so membership follows workloads rather than hand-maintained IP lists that drift as pods restart.
Also align decryption and no-decrypt rules to the same owner, since that is where the split caused this incident. Interim control: a monthly effective-policy review with a diff, signed off by both teams, until the single plane is in place.