Partner API With Duplicate Shipments, Out-of-Order Webhooks, and a Required Field

Duplicates. Two layers, because client discipline is not a guarantee.

• Business-level uniqueness first, because it works today with zero partner changes: a unique constraint on (partner_id, client_order_ref), with the create endpoint returning 200 plus the existing shipment when the reference already exists rather than 409. That alone would have prevented the 3,400 duplicates, since the partner was resending identical payloads.
• Protocol-level idempotency second: accept an Idempotency-Key header, persist key plus a hash of the request body plus the response with a 24-48 hour TTL, replay the stored response on retry, and return 422 if the same key arrives with a different body. Make it optional at first, then required for shipment-creating endpoints after a documented date.
• Fix the trigger, not just the symptom: the 502 came from a gateway timeout on a slow write. Retry loops are rational when the client cannot distinguish "failed" from "succeeded but the response was lost". Once idempotency exists, document retry guidance with exponential backoff and jitter.

Ordering and gaps. You cannot promise ordering over independent HTTP deliveries, so publish the metadata that lets consumers reconstruct it.

• Add a per-partner monotonically increasing sequence and an occurred_at timestamp to every event envelope. Consumers detect gaps by sequence and repair them through GET /v1/events?after=<sequence>, which is also the recovery path after downtime.
• Guarantee ordering per order by allowing only one in-flight delivery per order key on the sender side, and tell consumers to keep last-applied sequence per order and discard stale ones. Global ordering stays explicitly out of scope, which is honest and testable.
• Add a stable event_id and require dedupe on it; at-least-once is the only guarantee you can keep.
• Sign deliveries with HMAC over the raw body plus timestamp, with a documented public key endpoint, so consumers can reject replays and spoofing.

Ship the uniqueness constraint, ordering metadata, and replay endpoint server-side. Partner action is limited to consuming the new fields, and the replay endpoint is optional for them - that is what makes the change adoptable.

Rule one: never tighten validation inside an existing version. Making recipient.taxId required in v1 will break a subset of partners at an unpredictable time, and the failures will look like your outage. Instead:

• Add taxId to v1 as optional immediately, validate it when present, and return a warning in a response field for partners not sending it. In v2 it is required. Partners get a migration window where their existing traffic still works.
• For status, do not rename the field. Add statusDetail and statusVersion alongside the existing enum in v1, populate both, and let v2 emit only the new vocabulary. Consumers that map by exact string keep working; consumers that want the richer model can adopt early.

Versioning mechanics. URL major versioning is the right choice here given the partner population and their tooling; content negotiation adds a failure mode most of these integrators cannot debug. Add Sunset and Deprecation response headers, publish a written policy (12 months from v2 GA to v1 sunset, with a minimum of 90 days notice to any partner shown as active), and instrument usage per API key per version. Do not schedule the v1 sunset until the usage report shows zero remaining traffic excluding a named exception list.

SOAP holdouts. Do not build a parallel business implementation. Put a protocol-translation facade in front of the same domain services: the SOAP layer parses the envelope, maps the payload into the same internal command objects the REST layer uses, and maps domain errors into SOAP faults from the same error taxonomy. The facade is a mapping layer with its own contract tests and no business rules. Budget for it explicitly; a shared-core facade is roughly a week of work, while a second implementation is a permanent tax and a source of divergence.

Rollout. Ship v2 in dark mode to two friendly partners, compare responses against v1 for their traffic, then open it generally. Keep a contract test suite that runs every published example from the docs against staging on each deploy, so the documentation and the API cannot drift apart silently.

Why offset pagination loses rows. OFFSET 100000 LIMIT 100 makes the database count and discard 100,000 rows, so deep pages get steadily slower - that is why partners need 40-60 calls and hit rate limits. Worse, offset pagination is defined against a mutating sorted set. If an order is inserted ahead of the current offset between two calls, one row is skipped; if a row is deleted, one is repeated. Over a 2.1-million-row sync taking tens of minutes, both happen continuously. The missing orders are a correctness bug inherent to the design, not a partner mistake.

Keyset pagination.

• GET /v1/orders?cursor=<opaque>&limit=100, where the cursor encodes the last seen (created_at, id) tuple. The query becomes WHERE (created_at, id) > ($1, $2) ORDER BY created_at, id LIMIT 100, which is index-supported and constant-time regardless of depth.
• Sign and encrypt the cursor so partners cannot fabricate one and so you can change its contents without a version bump.
• Return next_cursor: null for the last page. Do not return a total count by default - it is a full scan; offer a separate /orders/count endpoint with caching for partners who genuinely need it.
• Keep offset working in v1 for a deprecation window, capped at a maximum depth (for example 10,000) with a clear error beyond it, so the worst behaviour degrades loudly rather than silently.

Rate limiting.

• Return Retry-After on every 429 and include X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset so a well-behaved client can self-pace.
• Limit per endpoint class rather than one flat bucket: reads get a generous budget, writes get a small one. A partner syncing orders should not be able to consume the shipment-creation budget.
• Consider token bucket with short bursts allowed, so a keyset sync that fires 20 sequential fast pages is not throttled as if it were abuse.
• Publish the limits in the docs with a reference backoff implementation. Most retry storms are clients doing the only thing they can think of because the server told them nothing.