Order Service Under Flash-Sale Load and Two Writers

Autoscale is a good fit for this shape - spiky, short, and predictable in time if not in amplitude.

Arithmetic (autoscale bills each hour at the highest RU/s reached in that hour, with a floor of max/10):

• Manual at 100,000 RU/s: ~100,000 × 24 = 2,400,000 RU-hours per day, paid whether used or not.
• Autoscale with max 100,000: the flash-sale hour bills at ~100,000, the other 23 hours bill at the floor of 10,000 → 100,000 + 230,000 = 330,000 RU-hours per day, roughly a seventh of the cost, while still absorbing the spike.

Caveats to state in the design:

• Max must exceed the peak. If you set autoscale max to 100,000 and the sale genuinely needs 100,000, you are at the ceiling and will throttle exactly as before. Set max to ~120,000 for headroom.
• Autoscale does not fix hot partitions. If the spike is concentrated on a handful of customer partitions (a single viral SKU sold to one account), autoscale adds account headroom but each physical partition still caps near 10,000 RU/s.
• Autoscale cannot be combined with the throughput you set on a shared-throughput database vs a container - pick one.

The configuration mistake that makes it useless: leaving the client retry policy as-is and using autoscale as an excuse to stop handling 429s. Autoscale scales in steps and takes time to react; a 20-minute spike that ramps in seconds will still produce 429s at the front edge. Keep bulk mode, keep x-ms-retry-after-ms-aware backoff, and alert on 429 rate rather than assuming the platform handles it.

Two layers, in this order.

1. Reduce the conflict at the data-model level: patch, don't replace. If each service owns distinct fields, they should never send a whole document. Use PatchOperations with Set on only the fields that service owns (pricing: /discount, /discountReason; fulfilment: /status, /carrier, /dispatchedAt). Patch is a targeted update, costs less RU than a replace, and eliminates the class of loss where service B overwrites fields it never read. This alone likely resolves the symptom, because the overwrite is caused by a read-modify-replace on a document shared by two owners.

2. Add optimistic concurrency where read-modify-write is unavoidable. Any field whose value is computed from the current value (a running total, a status transition guard) must be read, validated, then written with ItemRequestOptions { IfMatchEtag = readItem.ETag }. A concurrent change makes the write fail with 412 PreconditionFailed; the service re-reads, re-applies, and retries.

Avoiding the retry storm:

• Bound retries (3-5) with jittered exponential backoff, and on exhaustion fail the operation visibly rather than silently last-write-wins. A silent fallback reintroduces the bug you just fixed.
• Do not retry 412 as if it were a throttle; it is a lost race, not a capacity problem, and it needs a fresh read first.
• For high-contention counters (e.g. a per-customer running total touched by both services), stop fighting and use PatchOperation.Increment, which is applied server-side against the current value and has no read-modify-write window. ETag guards are for invariants, not for counters.
• If a status transition must be enforced (no Dispatched before Paid), put the guard in a transactional batch or stored procedure that reads and writes inside the same transaction, rather than relying on a client-side check between two round trips.

Atomicity breaks first, and the team may not have noticed. A transactional batch is atomic only within a single logical partition key and a single container, with at most 100 operations and a 2 MB payload. Moving line items to another container makes 'order plus 40 lines in one commit' impossible - the batch no longer spans both.

Options, in order of preference:

1. Keep one container with a type discriminator (order / line) keyed by /customerId, so the order and its lines share a logical partition and one batch can commit them. Watch the two hard limits: 40 lines is fine for the 100-operation cap, but the 2 MB batch limit is the one that bites when line items grow - and it argues against keeping the order document under 100 KB in the same item anyway.
2. Outbox pattern across containers. Write the order with a pendingLines marker in one batch, then a change-feed processor writes the line items idempotently (dedupe on orderId + lineNumber). This gives eventual atomicity with a visible intermediate state; the UI must be able to render 'order placed, lines syncing' rather than pretending it is instantaneous.

Second write region changes the consistency contract. Multi-region writes are incompatible with strong consistency; you drop to session, consistent prefix, or eventual. Two consequences:

• Conflict resolution becomes real. With the default last-write-wins policy on _ts, the fulfilment/pricing overwrite bug reappears across regions even after you add ETag guards, because ETags only catch conflicts the write can see. For orders, LWW on a whole document is the wrong semantics: either keep single-document ownership per region (route pricing writes to one region) or implement a custom conflict resolution stored procedure that merges field-level ownership and a monotonic version number rather than comparing timestamps.
• Conflicts are only surfaced for writes, and only until resolved. Monitor the conflicts feed; unresolved conflicts are invisible data loss that surfaces as a support ticket three weeks later.

Blunt recommendation: enable multi-region writes for read locality and write availability only after the field-ownership and conflict-resolution design is settled. Shipping it first will convert a reproducible bug into an intermittent, region-dependent one.