Multi-Cloud Kubernetes for Card Authorization Under a Hard Latency Budget
Meridian Payments is moving its card-not-present authorization service from two owned data centers to Kubernetes clusters in two public clouds, one primary region and one for failover. Peak load is 12,000 transactions per second and the p99 end-to-end budget is 40 milliseconds, of which roughly 9 milliseconds is currently consumed by network hops inside the data center. The platform is in PCI DSS scope, and clusters are shared by 40 internal tenants, several of which handle no cardholder data at all. Card and PIN cryptography today runs on Thales payShield hardware security modules in the primary data center. The target cloud region offers a general-purpose KMS with bring-your-own-key but no payment-grade HSM service. Segmentation today is security groups plus VLANs; there is no microsegmentation and no service mesh. The security operations team is six people with an outsourced 24x7 SOC. A service mesh pilot adds 3 to 5 milliseconds per hop and about 22 additional control-plane pods. The last external penetration test produced three critical findings, two of them lateral movement paths between tenants.
1. Design the segmentation and trust-zone architecture for the authorization service across both clouds. Where do you place enforcement, and how do you justify it against the 40-millisecond budget and the PCI scope?
Read the worked answer
Segmentation is a scope and blast-radius decision before it is a technology decision. The two critical findings were lateral movement between tenants; the cheapest durable fix is to stop co-locating the cardholder data environment with tenants that do not need it.
Trust zones: (1) cardholder data environment - authorization service, tokenization, key brokers; (2) a key management and crypto zone; (3) a shared-services zone (logging, CI/CD, monitoring, egress); (4) tenant workloads outside CDE scope. Zones 1 and 3 are separated by a controlled boundary with a default-deny policy and an explicit, minimal allow-list; zone 4 has no network path into zone 1 at all, only API calls through a published gateway.
Cluster topology. A dedicated cluster for the CDE in each cloud, not a shared 40-tenant cluster with namespaces. Namespace-level separation inside a shared control plane is a weaker isolation claim and makes the PCI scope argument much harder, which costs audit effort every year. The non-CDE tenants stay in a separate cluster with separate node pools and separate cloud identities.
Enforcement placement against the latency budget. Inline inspection is expensive per packet, so place it at zone boundaries and at ingress/egress, never per pod. Inside the node, use host-level policy and eBPF-based enforcement rather than a chained virtual appliance. East-west policy between pods is expressed as identity-based allow-lists enforced at the node and at the cluster network plugin, which adds negligible latency compared with a service-chain traversal. Egress from the CDE goes through a single controlled egress gateway with TLS inspection off for payment traffic (it is already mutually authenticated and encrypted) and full logging on.
Cross-cloud and failover. Interconnect the clouds through dedicated private circuits with encryption, not the public internet; replicate the CDE into the failover region with the same zone model rather than a reduced-control variant that nobody tested.
Measurement. Instrument the hop budget explicitly: publish a per-hop latency budget of the 40 milliseconds and treat any control that consumes more than 1 millisecond at p99 as requiring an architecture decision record with data, not a vendor claim.
2. The program wants to move all cryptography to the cloud provider's KMS with bring-your-own-key. As security architect, what do you decide about HSM strategy for card and PIN data, and how do you reason about the trade-offs?
Read the worked answer
Separate the key types before deciding. Data-encryption keys protecting stored cardholder data at rest are one problem; PIN blocks, PIN translation, and the issuer/master key hierarchy are a different one. General-purpose cloud KMS with BYOK is typically acceptable for the first category when the provider's HSM backing and key-use logging meet the standard the acquirer requires. It is generally not sufficient for PIN processing, where the requirement is a payment-grade HSM with the right firmware, key ceremonies, dual control, and split knowledge.
On the stated facts, the target region has no payment HSM service. That constrains the design, not the ambition. Options:
- Keep PIN handling where the payShields are and have the cloud authorization service call the on-premises HSM estate over a private, low-latency circuit. Viable for a first phase; the cost is a hard dependency on the data center for a cloud service, plus latency on the most latency-sensitive call, which must be measured against the 40-millisecond budget before commitment.
- Use a payment HSM service in a region that offers one, accepting the data-residency implications and the extra hop.
- Defer PIN-handling workloads to a later phase and migrate only card-not-present authorization paths that do not require PIN translation.
Recommendation: combine 1 and 3. Migrate the non-PIN authorization path now, keep PIN translation anchored to the existing HSM estate behind a private circuit with measured latency, and do not approve a design that terminates PIN keys in a general-purpose KMS.
Non-negotiables in any option: key ceremonies with dual control and split knowledge preserved in the new operating model (this is where cloud migrations usually fail, because the ceremony process is not migrated along with the technology), documented key custodian roles in the cloud provider's world, key-use logging into the same SIEM as the rest of the CDE, and a tested compromise-and-rotation procedure. If the program cannot describe who the key custodians are on day one in the target state, the design is not ready regardless of which HSM is chosen.
3. Decide between a full service mesh and native mutual TLS plus centralized policy for the CDE clusters. Show the reasoning, including what evidence you would require before committing.
Read the worked answer
Frame the decision as a trade between L7 policy granularity and operational load, measured against a hard latency budget and a six-person team.
What a mesh buys: consistent workload identity and mTLS across both clouds and both orchestrators, L7 authorization policy (method, path, caller identity), uniform traffic telemetry, retries and traffic shifting for failover. In a 40-tenant environment with a history of cross-tenant lateral movement, per-caller L7 policy and uniform identity are genuinely valuable.
What it costs: 3 to 5 milliseconds per hop against a 9-millisecond current internal network cost inside a 40-millisecond budget; 22 additional control-plane pods to patch, monitor, and upgrade in a PCI-scoped environment; and a control plane that becomes a Tier-1 dependency for authorization traffic, which a six-person team with an outsourced SOC will feel during every upgrade.
Recommendation: native mutual TLS for transport identity plus centralized policy enforcement at ingress, egress, and the zone boundary, with the mesh deferred pending measurement. Reasons: the highest-value authorization decisions here are zone-boundary and caller-identity decisions, which a policy engine at the gateway and at the node can enforce; the mesh's incremental value is mainly intra-zone L7 policy, where the tenant isolation problem has been solved structurally by putting the CDE in its own cluster; and three to five milliseconds per hop across a multi-service authorization path is a real fraction of a 40-millisecond p99 budget that is already partly consumed.
Evidence required before reversing this: a spike on a production-like path measuring p99 added latency at three or more hops, not a single-hop average; a measured control-plane upgrade and rollback exercise with the actual on-call team; and a demonstrated failover with the mesh control plane degraded, proving authorization traffic still flows. Record the outcome as an architecture decision record with a review date, because 'defer' is a decision with an owner and a trigger, not an omission. If the spike shows under 1 millisecond at p99 and the upgrade exercise is clean, adopt the mesh for the CDE clusters only.