Duplicate inventory reservations after timeouts and resubmits
An order-to-cash integration receives 5,000 orders/day from Salesforce through a REST trigger. For each order it calls Inventory Service POST /reservations, which has p99 latency of 8 seconds. The invoke timeout is 10 seconds, and the flow retries the invoke up to 3 times on any fault. When an instance ends in a fault, operations resubmits it from the monitoring console. Last month 37 units were oversold: reservations were created twice for 74 orders. The global fault handler sends one generic email to a shared mailbox for every fault, so nobody noticed in time. Non-recoverable errors such as HTTP 400 invalid SKU are retried the same way, adding about 30 seconds per order and drawing rate-limit responses from the partner.
1. Explain how the duplicates were created and identify the fixes in priority order.
Read the worked answer
Both retries and resubmits re-execute a call that may already have committed. A 10-second client timeout means 'I stopped waiting', not 'the server did nothing'; with p99 at 8 s, part of the distribution exceeds 10 s and completes server-side after the client gives up. The flow then retries up to three times, creating further reservations, and if the instance finally faults an operator resubmits the whole flow. That at-least-once path explains 74 doubled orders.
Priority fixes: (1) send an idempotency key derived from business identity (orderId plus lineId) and confirm the service honors it, or add a GET /reservations?orderId= status check before creating; (2) restrict retries to connection-level failures and HTTP 429/503 with exponential backoff and jitter - never retry 400/422, and never retry a timeout blindly; (3) forbid manual resubmit until reconciliation against the reservation service confirms the reservation is absent. Only then tune timeouts.
2. Redesign the error handling so business errors, transient errors and infrastructure faults are treated differently.
Read the worked answer
Make the classification explicit in the flow. Put a scope around the reservation call with a fault handler that branches three ways. Business errors (400/422 invalid SKU) go to a dead-letter store - a table or queue holding order id, payload reference, error code and timestamp - with no retry, a clear status returned to Salesforce, and notification to the order owner. Transient errors (timeouts, 429/503) get a bounded retry with backoff inside the scope, then land in the same store tagged 'needs reconciliation' rather than being silently dropped. Infrastructure faults propagate to the global fault handler, which must raise an actionable alert instead of emailing a shared mailbox.
Add a business identifier on the trigger so instances are searchable by order id, and alert on rates rather than single events: more than 5 reservation faults in 10 minutes, or dead-letter depth above 50, routed to OCI alarms and on-call. Return a correlation id to Salesforce for support tracing. Never use 'ignore error' inside a for-each around an inventory or financial call - silent continuation is how the 37 units shipped.
3. Should the 10-second timeout be raised, and what must stay true if you change it?
Read the worked answer
Raise it, but as a secondary measure. A 10 s cutoff against p99 8 s amputates a meaningful tail of successful-but-slow calls, so 12–15 s is more appropriate - provided two invariants hold. First, the invoke timeout must remain shorter than the caller's own timeout, so this flow produces a clean, classified failure instead of being abandoned from upstream. Second, the retry budget must stay small, because worst-case duration is attempts × timeout: three attempts at 15 s already exceeds most order SLAs.
Better still, ask whether the reservation can be asynchronous (202 plus a poll or callback). That removes the timeout ambiguity entirely and makes idempotency trivial, because the flow polls a known operation id instead of guessing whether a timed-out POST committed. If async is unavailable, keep 12–15 s, cap retries at two, and rely on the idempotency key - not on timing - as the real protection against duplicates.