Duplicates, a five-attempt retry, and an 800,000-message backlog

Diagnosis. There are two retry mechanisms stacked on each other: the HTTP retry inside the flow and the MQ redelivery after maxRedeliveryAttempts. A timeout is an unknown outcome, not a failure - the ERP may have committed before the response was lost. Retrying an unknown outcome without an idempotency key guarantees duplicates whenever the commit succeeded. MQ gives at-least-once delivery, so the consumer is also required to be idempotent regardless of the HTTP layer. Both mechanisms retry precisely the case where retrying is unsafe.

The fix - an idempotency key and a state machine.

1. The order producer stamps a stable business key (orderId plus version) into the message properties; it must be generated upstream, not by the consumer.
2. Before calling SAP, the consumer performs an atomic check-and-set in an idempotency store (Object Store or a database table with a unique constraint): insert key with state IN_PROGRESS. If the insert conflicts and the stored state is COMPLETED, return the stored response and acknowledge the message - this is the duplicate case. If the state is IN_PROGRESS, treat it as a concurrent duplicate and do not call SAP.
3. On success, record COMPLETED with the SAP document number.
4. On an unknown outcome, leave the record in an indeterminate state and route to reconciliation rather than blindly retrying.

What cannot be atomic: the idempotency write and the SAP commit are in two systems, so there is no distributed transaction and you should not build one. Accept that and close the gap with reconciliation.

Prefer the strongest option available: if SAP exposes an idempotent interface keyed on an external order ID (an idempotent BAPI or an API with a request key), use it - the ERP then enforces uniqueness and the store becomes a fast path rather than the only line of defence.

Classify before you retry. Retryable: connection refused, connection reset before any bytes were sent, 429 and 503 with Retry-After, 5xx from a gateway. Non-retryable: 400 validation or mapping errors, 401/403, and SAP business rejections such as credit block or unknown material - retrying these five times only delays the inevitable and floods the queue. Unknown: read timeouts and 5xx that may follow a commit. These are the dangerous ones and must go to reconciliation, not to a retry loop.

Implementation. Use an until-successful scope with a small attempt count (3) and exponential backoff with jitter rather than a fixed 3-second interval, so a recovering ERP is not hammered by synchronised retries. Route exhausted transient failures to a dead-letter queue carrying the original message plus error type, attempt history, correlation ID and timestamp. Route business rejections to a separate business-error queue with an owner in operations, because they need a human, not a retry. Keep MQ redelivery as a last-resort safety net at a low count - it must not be the primary retry strategy.

Reconciliation. A scheduled job compares source order counts and amounts against SAP-posted documents for each window, flags unmatched items, and resolves the indeterminate set by querying SAP for the external order ID. Build a replay tool that re-submits a dead-lettered message with its original idempotency key, so replay is safe by construction.

Operate it. Alert on dead-letter depth and age, on reconciliation variance above a threshold, and on indeterminate-state count. The 0.3% manual reconciliation exists because nobody was watching; the target is zero manual work with an alert when it is not zero, not silent success.

Find the actual bottleneck before scaling. At peak, 3,000 messages per minute is 50 per second. With 4 workers at, say, 16 concurrent consumers each, you have 64 concurrent calls; if SAP latency rises from 200 ms to 10 seconds, throughput collapses regardless of worker count, and adding workers simply increases concurrent pressure on the system that is already struggling. Scaling workers is the wrong lever when the downstream is the constraint - it converts a queue backlog into an ERP outage.

Bound the queue instead of growing it. Apply backpressure at ingress: accept up to a defined queue depth and then reject new work with 429 and Retry-After, so senders retry on their own schedule rather than being told 'accepted' for work that will run hours later. Shedding load early is more honest than an 800,000-message backlog.

Separate the lanes. Put latency-sensitive real-time orders and bulk or batch traffic on distinct queues with distinct worker allocations and, if needed, distinct ERP connection pools. Reserve a share of capacity (for example 20%) for high-value or time-critical orders so a bulk surge cannot starve them.

Fix the webhooks. Status webhooks must fire from the outcome of the ERP post, not from ingestion. Emitting 'order received' at the queue boundary and 'order confirmed' on commit separates the two, and a status API the customer can poll removes the dependency on a webhook arriving on time. If the contract requires webhook delivery, publish from the consumer after commit, with the idempotency key as the event ID so consumers can deduplicate.

Observe and autoscale on the right signal: queue depth and consumer lag, not CPU. Cap autoscaling below the ERP's safe concurrency, and get SAP's capacity limit in writing - that number, not the queue, defines your ceiling.