Retry storms, duplicates, and a rate-limited payment provider

The duplicates are a direct consequence of retrying everything without an idempotency key. During a 30-minute outage the PSP returns 503 on the charge POST; the fixed 1-second, three-attempt retry fires, some attempts land once the PSP recovers, and each successful retry creates a new charge because the PSP has no way to know it is the same logical payment.

Design:

1. Send a deterministic idempotency key - Idempotency-Key: {orderId}:{attemptBatchId} derived from the order, not a fresh GUID generated per attempt. The PSP deduplicates on it, so a retry after an ambiguous success is safe. This alone converts retries from dangerous to harmless for the charge endpoint.
2. Classify before retrying. Retry 429, 503 (when the method is idempotent or keyed), and socket/timeout failures. Never retry 400/401/402/422 - they will fail identically and burn the rate limit. A timeout is the ambiguous case and is only safe because of item 1.
3. Honour Retry-After rather than a fixed delay: add a Polly policy that reads the header, and use exponential backoff with jitter so 400 concurrent invocations do not synchronise into a thundering herd.
4. Add a circuit breaker that opens after a handful of consecutive 503s and stays open for the expected outage duration, so during a 30-minute window you fail fast to a queue or to a 'payment pending' state instead of hammering a dead endpoint. The retry policy must sit inside the breaker, not outside it.
5. Reconcile. Use the PSP's list/settlement endpoint to detect charges created by ambiguous attempts, and make the local ledger write idempotent on the PSP charge id.

Also note: the charge POST is not idempotent by HTTP semantics, so 'retry on 503' is only correct once item 1 exists.

Two different limiters, and conflating them is the classic mistake here.

Inbound, per partner: API Management rate-limit-by-key (or the newer azure-openai-token-limit-style policies for token budgets) keyed on the subscription or a partnerId claim, with calls: 100 and renewal-period: 60 and a counter key. That caps partners, gives them a clean 429 with retry headers, and keeps the PSP key out of their hands entirely. It says nothing about protecting the PSP, because ten partners at 100/min still equals 1,000/min outbound.

Outbound, to the PSP: one shared budget across all instances, which a per-instance in-process counter cannot enforce. Options:

• Queue-based load levelling: partners enqueue payment jobs; a worker drains at a controlled rate. This also gives you the natural place to park work during a 503 outage - which you need anyway.
• A distributed token bucket in Redis (atomic INCR with a 60-second TTL per key, or a Lua script), checked before every outbound call. Precise, but adds a dependency and a failure mode on the hot path.
• AFunction with maxConcurrentCalls and session-based pacing so the concurrency ceiling times the call latency stays under 100/min. Simple, approximate, and it degrades gracefully.

Order of preference: queue plus a paced consumer, because it also absorbs the maintenance window; Redis only if you need exactness. APIM alone is insufficient - say so explicitly.

A new HttpClient per call means a new SocketsHttpHandler per call, so connections are never reused: each call pays a TCP+TLS handshake, and the disposed handlers leave sockets in TIME_WAIT until the OS exhausts ephemeral ports under load (SocketException: Only one usage of each socket address...). On a Consumption plan that fans out to dozens of instances, you also multiply the handshake cost against a partner that is already rate-limiting you - the handshakes consume part of the same 100/min budget headroom.

Correct pattern:

1. IHttpClientFactory registered in DI, with a named client carrying the PSP base address, timeout, and a DelegatingHandler chain for the idempotency header, retry, and circuit breaker. The factory pools and rotates handlers, so you get connection reuse without the stale-DNS problem of a single static client.
2. Set SocketsHttpHandler.PooledConnectionLifetime (a few minutes) if you need explicit DNS refresh control, and set MaxConnectionsPerServer deliberately so you do not open more concurrent connections than the rate limiter will allow anyway.
3. Set a per-attempt timeout on the client shorter than the PSP's own timeout, and let the Polly Timeout policy own it, so a hung request releases its slot for the next queued payment.
4. Never resolve the client from DI as a singleton HttpClient without pooling, and never dispose a factory-created client - the factory owns the handler lifecycle.

The judgement: this is not a micro-optimisation. Socket exhaustion presents as random failures under exactly the load conditions that produced the duplicate charges, so fixing the retry policy without fixing the handler leaves the incident reproducible.