Retry Storms and Duplicate Invoices in the Orion Integration

The duplicates are a design defect, not a tuning problem. The chain is:

• The middleware sends an invoice request. The BAPI takes longer than 20 seconds, so the caller sees a timeout.
• The timeout is ambiguous: the ERP may still be processing. Without an idempotency key, the retry creates a second invoice rather than returning the first result.
• Retrying every 30 seconds while the original call is still running multiplies concurrent work on a BAPI that is already slow under load. That is a retry storm, and it explains why the duplicate rate is highest during peak.
• The queue growth is the same mechanism at outage scale. During the carrier outage, every message fails, every failure triggers three retries, and nothing stops intake. A 240,000-message queue in a 90-minute window is consistent with retry amplification plus no load shedding or circuit breaker.
• There is no dead-letter path, so poisoned messages are retried forever and healthy messages queue behind them.
• The peak rate of 12,000 orders per hour is only about 3.3 per second, well under the 400 requests per second API limit. The bottleneck is the 20-second BAPI and the retry policy, not raw throughput. That distinction matters: buying more capacity would not fix this.

Raising retries to five and shortening to 5 seconds makes every failure mode worse. It increases concurrent load on the BAPI, shortens the window in which a slow success can complete, and creates more duplicates.

Move from synchronous retry-based calls to an asynchronous, idempotent, reconcilable flow.

• Idempotency key per business event, derived from order number plus line plus revision, carried on every attempt and enforced by a deduplication table in ERP or in the middleware before the BAPI call. This converts an ambiguous timeout into a safe retry.
• Asynchronous request-reply: middleware posts the order, ERP returns an acknowledgement and a correlation ID, and the result is posted to a callback or polled. No caller waits 20 seconds on a BAPI.
• Exponential backoff with jitter, capped, and a maximum attempt count. Jitter prevents synchronized retry waves.
• Circuit breaker that stops sending to a failing endpoint and sheds or parks load, plus rate limiting aligned to the BAPI's real capacity rather than the API gateway's 400 per second.
• Dead-letter queue with alerting and a replay tool, so failures are visible and recoverable instead of blocking the queue.
• Outbox pattern on the cloud side so an order event is never lost between database commit and publish.
• Scheduled reconciliation comparing order counts and amounts across both systems, with an exception report. This is the control finance actually needs, because it detects duplicates that slip through regardless of design.
• Batching for invoice posting where the BAPI supports it, to reduce call volume during the peak window.

Each element addresses a specific observed failure: duplicates (idempotency, reconciliation), storms (backoff, circuit breaker, rate limit), and drain time (dead-letter, load shedding).

Commit to a measurable control, not a slogan. "Zero duplicates" is achievable only with idempotency plus reconciliation plus a freeze on uncontrolled retries; it is not achievable by tuning alone in 12 days.

Proposed commitment to finance: from the cutover of the idempotency change, any duplicate invoice is either prevented at the deduplication layer or detected by the daily reconciliation before posting to the ledger. Provide a duplicate-detection query and a named owner for the exception report.

Testing in the window:

• Replay a production-like 90-minute peak at 12,000 orders per hour with the new pattern and measure duplicates, latency, and queue depth.
• Force BAPI timeouts deliberately and verify that retries do not create a second invoice.
• Simulate a 30-minute carrier outage and confirm the circuit breaker opens, the queue stays bounded, and drain time is acceptable.
• Test the dead-letter replay path and confirm no message is processed twice.
• Run reconciliation against a seeded data set containing known duplicates and confirm all are detected.
• Define rollback: if duplicate rate exceeds an agreed threshold during the first two days, revert to the previous flow with retries disabled and post invoices through a controlled batch.

Sequence matters. Disable the aggressive retry policy immediately even before the full pattern is built; that alone reduces duplicate creation and queue amplification. Tell finance plainly that the 5-second retry proposal would increase both duplicates and outage impact, and that the 12-day plan addresses prevention and detection rather than assuming the problem away.