Cosmos DB hot partition under a fixed RU budget

This is the classic hot-partition signature. Provisioned RU/s is divided across physical partitions, and a single logical partition is pinned to one physical partition: it can never consume more than the throughput of that one partition, no matter how much headroom exists elsewhere. The wholesale tenant's ~17 GB is also brushing the 20 GB logical partition ceiling, so it has to be split regardless of throughput.

So the 60% figure is an average hiding a saturated partition - the metric to check is Normalized RU Consumption per partition key range, and the diagnostic log's PartitionKeyRangeId on the 429s.

Options:

1. Hierarchical partition keys (/tenantId, /orderId) on a new container. This is the intended fix for exactly this shape: it spreads the large tenant across physical partitions while preserving the tenant-scoped query pattern. Migration is a new container plus a copy.
2. Synthetic key (tenantId + hash(orderId, N)) if you need something the SDK and older tooling handle identically. Cost: every read now needs the suffix computed, so point reads by tenantId alone become fan-out queries.
3. Do nothing structural and switch to autoscale - rejected. Autoscale bills for peak and does not route around a saturated partition; it just raises the bill while the 429s continue.

For the two-hour cutover: stand up the new container, backfill with the change feed (or ADF for the bulk copy), dual-write or freeze writes briefly, then flip the connection string in configuration. The change feed gives you a continuous catch-up so the write freeze is minutes, not two hours.

A query that omits the partition key is always cross-partition, and the RU cost scales with the number of physical partitions touched plus the documents scanned. Point reads cannot help unless both id and the partition key value are supplied - Cosmos cannot do a 1-RU read without knowing where the document lives.

Three viable answers, in order of preference:

1. Materialised lookup container maintained by the change feed. A second container keyed /orderId with id = orderId, holding only the fields support needs. A change-feed processor function (or the built-in Azure Functions Cosmos trigger) projects every write into it. Support gets a 1-RU point read.
   • Cost: one extra function app running continuously plus duplicated storage for the projection. Consistency: eventual, typically sub-second, but the portal must not use this container for authoritative writes.
2. Make the portal supply tenantId. Cheapest by far - zero storage, zero code. Requires a UI change and a lookup path from the order to the tenant; often the real fix because support usually has the customer in front of them.
3. Add orderId to a synthetic partition key. Rejected here: it would break the tenant-scoped catalog queries that dominate traffic, and it does not fit the two-hour cutover.

If a projection is built, also consider whether the support container should be a separate container with lower throughput rather than sharing the catalog's 10,000 RU/s - otherwise the month-end reporting spike starves support lookups again.

At 60% average utilisation you are not actually short of RU/s - you are spending it badly. Start with measurement, not with buying:

1. Turn on diagnostic logs to Log Analytics and rank requests by RU charge grouped by operation type, container, and caller. The 24-hour and 7-day Total Request Units metric tells you the trend; the logs tell you who. The month-end aggregation is the obvious suspect and the one worth proving.
2. Move reporting off the transactional container. Synapse Link's analytical store keeps a columnar copy that Synapse queries without consuming any RU/s. This directly removes the month-end spike for zero RU cost - the trade-off is extra storage cost and a schema/analytics-store enablement step, so confirm with finance that storage is cheaper than RU/s (it usually is, by a wide margin). A nightly export to Parquet in ADLS queried via serverless SQL is the lower-commitment alternative.
3. Cache the hot catalog reads. The catalog is read-heavy and tolerates seconds of staleness. APIM response caching or a Redis-backed read-through removes a large share of RU-charging point reads entirely.
4. Right-size the throughput mode last. After 1-3, if the remaining load is spiky, autoscale may cost less than a manual 10,000 RU/s floor. Do not lead with this.

The judgement being tested: 429s do not automatically mean "buy more RU/s", and a fixed budget is a prompt to look for a design change rather than a capacity change.