One resource or three: standing up a shared Azure AI platform
Contoso's platform team of two engineers must host three workloads in West Europe: a Document Intelligence pipeline at roughly 40,000 pages/month across 11 suppliers, an Azure OpenAI chat workload at a sustained 120,000 TPM, and Azure AI Vision tagging about 900,000 images/month. Security mandates private endpoints only, with no public egress from the spoke VNet. Finance wants monthly cost attributed per workload. Legal requires that a leaked key for the chat workload must not be usable to read supplier document content. The chosen chat model has no capacity in West Europe and is only deployable in Sweden Central. The team's stated goal is the fewest possible resources to administer.
1. Recommend a resource topology and justify it against the team's 'fewest resources' preference.
Read the worked answer
Decision: two resources, not one and not three.
- Key isolation is a hard constraint. A single multi-service Azure AI Services resource issues shared keys; anyone holding the chat key could call Document Intelligence on the same resource. Legal's requirement therefore forces at least two resources. RBAC alone is not sufficient if the application authenticates with keys.
- Region is a hard constraint. An Azure AI Services resource is regional. Because the chat model is only available in Sweden Central, that resource must live there; the document and vision traffic should stay in West Europe for data residency and latency.
- So: Resource A = multi-service (West Europe) for Document Intelligence + Vision. Resource B = Azure OpenAI (Sweden Central).
Trade-off to surface: the team gains one extra resource to administer but also gains cleaner chargeback (per-resource cost instead of tag-based splitting) and independent private endpoints. Three separate resources adds no further benefit - Document Intelligence and Vision share no isolation requirement.
Also flag: cross-region calls from West Europe to Sweden Central add latency and create a data-residency question that Legal, not Engineering, must sign off on.
2. How should the team plan for the 120,000 TPM sustained chat load, and what happens at 3x peak?
Read the worked answer
Reason about quota before architecture.
- TPM quota is scoped to subscription + region + model + deployment type. A 120k TPM sustained figure cannot be assumed against default quota; the team must file a quota increase request per model in Sweden Central and confirm the approved ceiling.
- Sustained vs burst: sustained load suggests Provisioned Throughput Units (PTU) for predictable latency and cost, with a standard (pay-as-you-go) deployment as overflow. Pure standard deployment at 3x peak will return 429s.
- For the 3x peak: create two or three chat deployments of the same model and load-balance across them client-side (round-robin with retry/backoff on 429). Retry with exponential backoff, and cap per-request max_tokens so a single request cannot consume a large share of the window.
- Design the failure path: a queue plus graceful degradation ('please retry') is preferable to silent truncation. Measure token consumption per request, not requests per second - TPM exhaustion is usually driven by large prompts, which links directly to over-large retrieval contexts.
3. Keys must be rotated with zero downtime and the chat workload must never be able to read document content. What identity and rotation design do you specify?
Read the worked answer
Eliminate keys rather than rotate them well.
- Disable local (key) authentication on both resources and use Microsoft Entra ID authentication with managed identities. Each workload gets its own identity and a role assignment scoped to the one resource it needs: the chat identity gets Cognitive Services OpenAI User on Resource B only; the extraction identity gets Cognitive Services User on Resource A only. This makes the 'leaked key' scenario structurally impossible, not procedurally unlikely.
- If a stakeholder insists on keys (some SDK paths or third-party tools), keep two keys per resource, store them in Key Vault, have apps read via managed identity, and rotate on a schedule: regenerate key2, let apps refresh over a window longer than their cache TTL, then regenerate key1. Rotation is only zero-downtime if the client refresh interval is shorter than the rotation gap.
- Verify the private path: with public access disabled, Key Vault and both AI resources need private endpoints reachable from the spoke; add a private DNS zone per resource. A rotation design that depends on a public Key Vault endpoint will fail closed in production.