Hot Tenant on a Shared Shipments Container
Northwind Freight runs a multi-tenant SaaS on Azure Cosmos DB for NoSQL. All tenants share one container, shipments, with partition key /tenantId and 120,000 RU/s provisioned manually. Tenant Contoso generates 45% of all traffic; during its nightly load the account returns thousands of 429s while other tenants consume only 5-40 RU/s each. Application point reads are WHERE c.tenantId = @t AND c.shipmentId = @s. A platform dashboard runs SELECT * FROM c WHERE c.status = 'Delayed' ORDER BY c.createdAt DESC across all tenants every 30 seconds. Contoso's logical partition holds 14 GB and grows 3 GB per month. The client team can change which parameters it sends but cannot remove any predicate from the point-read queries.
1. Which partition key strategy removes both the hot-partition throttling and the per-tenant 20 GB ceiling without breaking the existing point reads? Justify against the physical limits, and state one query pattern that gets worse.
Read the worked answer
Recommendation: a hierarchical partition key of /tenantId then /shipmentId.
Why it fixes both problems:
- Throughput spread. Today 45% of 120,000 RU/s is aimed at one logical partition, which sits on one physical partition. A physical partition is capped at ~10,000 RU/s, so Contoso is throttled regardless of how much throughput the account owns. Splitting on
(tenantId, shipmentId)scatters Contoso's load across many logical and physical partitions. - Storage ceiling. The 20 GB limit applies per partition key value. With a single-level key, Contoso is one value and is 6 GB from the wall. With the leaf level
(tenantId, shipmentId), the limit applies per shipment, so the tenant is no longer capped at 20 GB. - No query rewrite. The point read supplies both
tenantIdandshipmentId, so after the change it is still a single-partition, ~1 RU point lookup. Nothing the client sends has to change.
Rejected alternative: a synthetic key such as tenantId + hash. It would spread writes but the client does not know the hash, so the predicate c.shipmentId = @s could no longer target the partition and every point read would fan out.
What gets worse: a query that filters on tenantId alone (for example a tenant-scoped list view) now becomes a partition-prefix query that fans out across every shipment partition for that tenant, costing RU proportional to partition count.
Operational note: partition key is immutable, so this needs a new container plus a change-feed backfill and a cutover. A hierarchical key allows at most three levels.
2. The dashboard query runs every 30 seconds across all tenants and competes with order traffic. How would you serve it, and what edge case in your design would bite you?
Read the worked answer
The query is a cross-tenant scan filtered on status and sorted on createdAt; there is no index that can avoid reading every delayed shipment, and running it every 30 seconds spends the same RU budget that orders need.
Recommendation: a change-feed materialized view. A change-feed processor projects each shipment into a small shipmentStatus container partitioned by /status, carrying only shipmentId, tenantId, createdAt. The dashboard becomes a single-partition query with a composite index on (status ASC, createdAt DESC) and costs a handful of RU.
The edge case: deletes. The change feed in latest-version mode does not surface deletes, so a shipment that is deleted (or purged by TTL) would linger in the view as a phantom 'Delayed' row and the dashboard would drift. Mitigations: use soft delete (a isDeleted flag) and let the processor remove the projection, or subscribe to the 'all versions and deletes' change-feed mode and handle delete events explicitly.
Alternative if the team wants zero extra containers: enable the analytical store and query it through Synapse Link or Fabric mirroring. That fully isolates the read from the OLTP RU budget, but adds cost and a sync-latency floor, which may be acceptable for a 30-second dashboard but not if someone later asks for sub-second refresh.
Do not simply add a composite index on the existing container and keep firing the query every 30 seconds - the index makes each execution cheaper but the fan-out across all partitions remains.
3. An engineer proposes doubling provisioned throughput to 240,000 RU/s to stop Contoso's 429s before the redesign ships. Will that help?
Read the worked answer
Barely, and possibly not at all. Provisioned throughput is a budget for the whole account; it does not raise the ceiling of any individual physical partition, which is ~10,000 RU/s. Contoso's nightly burst is concentrated in one logical partition, so it can consume at most what its hosting physical partition is given. Adding RU/s grows the number of partitions and the account's headroom, but a single hot key cannot outrun its partition.
What doubling will do: make the other 55% of tenants look healthier while costing twice as much, and mask the real symptom until Contoso's growth adds another 3 GB and the split logic reshuffles partitions again.
Legitimate short-term levers while the hierarchical key is built:
- Ask the client to spread the nightly load - reduce
MaxDegreeOfParallelism, or stagger the batch across the hour - so the burst does not exceed partition capacity. - Enable bulk mode in the SDK and handle 429s with a retry policy that honours
x-ms-retry-after-ms; the failures may be amplification rather than a hard cap. - Cap the batch job's throughput deliberately (client-side rate limiter) so it degrades predictably instead of starving other tenants.
Keep the throughput increase out of the plan unless measurements after the partition-key change show the account genuinely short on aggregate RU/s.