Global Support Assistant: Model Routing Under Latency and Residency Constraints
Northwind Bank runs a customer support assistant in 14 countries. Peak load is 22,000 requests per minute, with a contractual p95 first-token latency of 1.6 s and a monthly inference budget of USD 85,000. The assistant uses a retrieval layer over 3.1 million policy chunks in BigQuery and Cloud Storage. Regulators require EU customer prompts and retrieved snippets to remain in EU regions, with CMEK and VPC Service Controls. The current design sends every request to a single large Gemini model in us-central1. At peak, 31% of requests exceed the latency target, quota errors appear in three regions, and spend is 40% over budget. Product wants one model and one global endpoint to simplify operations. Security rejects any shared project that mixes EU and US data. The team has four weeks to propose a target architecture and a measurement plan before a production review.
1. Recommend a target serving and routing architecture. Explain the tradeoffs and whether a single global endpoint is appropriate.
Read the worked answer
Reject the single global endpoint. Residency is a data-flow constraint, not a documentation problem: an EU prompt and its retrieved snippets cannot be processed or stored in us-central1 under the stated CMEK and VPC-SC controls. A global endpoint also concentrates quota and adds cross-region round-trip time to a 1.6 s p95 target.
Target architecture. Deploy regional Vertex AI endpoints in at least the EU and US, each inside its own VPC-SC perimeter and project boundary, with regional BigQuery datasets and Cloud Storage buckets for retrieval. Put a global external Application Load Balancer in front for routing, but route EU traffic only to EU backends. Use a small model for intent classification and simple policy answers, and a large model only for complex cases; add a regional semantic cache. Reserve quota per region and alert on 429 responses.
Tradeoffs. This adds operational overhead and duplicate model deployments, but it is the only option that satisfies residency while reducing latency and large-model spend. A single global endpoint would be simpler but non-compliant and likely slower at peak.
2. Define the measurement plan that would prove the architecture meets latency, cost, and compliance targets before the production review.
Read the worked answer
Measure separately by region and by model tier, because averaging hides the EU and peak-hour failures. Track p50/p95/p99 time to first token and end-to-end latency, 429 and timeout rates, cache hit rate, model-mix percentage, input/output token counts, and cost per request and per resolved issue. Compare against the 1.6 s p95 and USD 85,000 budget with headroom for the 22,000 rpm peak.
Run three evidence streams: a load test at 1.2x peak with realistic prompt lengths, a 5% shadow or canary of the new router, and a failover game day that removes one regional backend. Acceptance criteria should include no cross-region EU data flow, no quota exhaustion at 1.2x peak, and cost per resolved issue below the current baseline.
Compliance evidence should include VPC-SC perimeter logs, CMEK key-usage records, data-location inventory, and a sampled trace showing EU prompts and snippets never leave the EU. Cost alone is not enough; a cheaper architecture that fails residency or p95 is not acceptable.
3. Product proposes replicating all 3.1 million policy chunks into every region. Evaluate this proposal and propose an alternative.
Read the worked answer
The proposal is over-broad. Replicating all chunks, including EU policy text, into US storage likely violates the residency requirement and creates a consistency problem: 3.1 million chunks updated frequently would need cross-region synchronization, and stale copies would degrade answer correctness. It also multiplies storage and embedding cost without solving quota or model latency.
Alternative. Classify chunks by residency and sensitivity. Keep EU-origin or EU-restricted chunks only in the EU dataset; keep global, non-sensitive policy metadata in each region if allowed. Route each query to the region that owns the retrieved content, and store embeddings in the same region as the source text. Maintain a small global catalog of chunk IDs, titles, and region pointers only if it contains no restricted content. Use BigQuery authorized views or regional datasets to enforce boundaries.
Validate retrieval recall and citation accuracy per region on a stratified sample. Measure cross-region pointer lookups and cache hit rate. If a global query needs an EU chunk, the correct behavior is to route the request to the EU backend, not to copy the chunk to the US.