NorthPeak Retail: latency, residency, and the cost of active-active

Order the constraints: residency is hard, write latency is soft, read latency is the real problem. Solve in that order.

1. Residency eliminates (b) as stated. Spanner multi-region instances replicate every row to every region in the configuration. A row in nam-eur-asia1 is stored in the US. There is no per-row placement. So legal alone forces at least two instances: EU data in an EU-only instance (europe-west1 regional or eur3), non-EU data in a US instance. Option (b) is only viable after you have already done (c).
2. Reads dominate, so cache and replicate reads, not writes. 85% of 14,000 QPS is ~11,900 read QPS. A 620 ms p99 from Frankfurt is round-trip plus query. One cross-region hop from us-central1 is roughly 90-140 ms RTT, which on its own would put p99 near 300 ms with no headroom. Local read replicas plus an application cache in front of the hot price/inventory keys remove the hop entirely for reads.
3. Writes are only 1,900/s and are latency-tolerant. A 150-250 ms cross-region commit for 15% of traffic does not breach a 300 ms p99 budget if the write path is asynchronous at the edge (accept, queue, confirm) or if writes are routed to the tenant's home region.
4. So the answer is (c), implemented as data-domain sharding with a routing tier, where the EU shard happens to satisfy legal and the US shard happens to satisfy latency, plus read replicas and caching for the read-heavy global path. (a) alone fails because it leaves 38% of users at 600 ms.

Cost check against the 2.4x envelope: you are adding one regional Spanner (or a Cloud SQL read replica pair) per additional region plus a cache tier, and keeping the existing tier as the write leader for its shard. That is roughly a doubling, not a 10x. A full active-active Spanner deployment across three regions with the node count needed for 14k QPS is the version that blows the envelope.

What I would explicitly not do: last-write-wins replication of inventory counts across regions. Two stores selling the same SKU during a network partition produce a wrong stock number, and that is a business incident, not an availability incident.

Split the data by whether the field is identifiable, then enforce the split at the infrastructure layer - design intent alone will not survive the audit.

• Authoritative EU records live only in the EU instance. The EU region's service account is the only identity with spanner.databases.read on it.
• Derived, non-PII read models (price, availability count, SKU attributes) are published globally via Pub/Sub into regional read stores or a global cache. The global path never touches raw EU rows.
• Request routing at the edge. Global external Application Load Balancer with URL/header routing plus a regional affinity key moves EU-origin requests to europe-west1. Anything requiring raw EU data terminates there.
• Enforcement, not convention:
  • VPC Service Controls perimeter around the EU project set, with the US projects outside it and no ingress rule allowing them. This is what makes 'the US service cannot read the EU database' true even if someone later grants IAM by mistake.
  • IAM Deny policies on the EU Spanner instance for all non-EU principals, which survive future IAM grants.
  • Regionally scoped log routing: EU request logs go to a europe-west1 log bucket with CMEK, with an exclusion on the _Default sink. PII leaking into global logs is a residency breach just as much as leaking into a US table.

Why not VPC-SC alone: it governs Google API access from inside a perimeter, but it does not stop a workload that legitimately lives in the perimeter from copying data out through an approved egress path. Pair it with data-level controls and the routing design.

Failure mode to test: a global dashboard that aggregates EU sales for a CFO report. That query must run as an aggregation against the derived model, not as a cross-instance join over raw rows. Add a check in CI that fails if any query plan references the EU instance from a non-EU service account.

State the kill criteria before you build, because after 90 days sunk cost will argue for you.

I would reverse if any of these hold at the 90-day review:

1. Write conflict rate above ~0.5% of writes on the shared entities (price overrides, stock adjustments). That means the shard key is wrong or the business genuinely has cross-region writes on the same row. Either way, the conflict-resolution code is now load-bearing and the team of 11 does not have the headroom to own it.
2. p99 in the EU region above 300 ms for more than 1% of minutes. This is the entire justification for the extra region. If caching and local reads did not get us there, the design is paying multi-region cost for single-region latency.
3. Any incident where the two shards disagreed and a human had to reconcile data. That is a new class of incident that did not exist before, and the on-call runbook for it will be written by someone at 3 a.m.
4. Error budget consumption above 50% in a quarter attributable to the multi-region control plane (routing failures, cache invalidation storms, asymmetric partitions).

The fallback, spelled out now so it is a real option: EU shard stays where it is (it must, for legal), US shard consolidates into a single regional primary with a cross-region read replica for the APAC path, and the global LB routes by affinity. RTO becomes a documented 10-15 minutes of manual failover rather than seconds. At 38% non-US traffic and 85% reads, that costs roughly half the cross-region footprint and preserves the p99 target, because the p99 target was always a read problem.

Why bother writing this down: the CFO asked for a written justification for outage exposure. This is that document, and it also prevents the team from defending a design that stopped paying for itself.