The Integration That Duplicates Orders on Retry
An order process calls an external fulfillment API at the end of the flow. The integration has a 30-second timeout and is configured to retry 3 times on failure. During a network blip, the API received the order but the response timed out; the process retried and the API created the order three more times. Downstream, 240 duplicate orders shipped before anyone noticed. The API has no idempotency key support today, but the vendor says they can add one. The process also has no record of which orders were sent, only that the step completed or errored.
1. Explain why the retry produced duplicates and what design changes prevent it, given the API lacks idempotency today.
Read the worked answer
Why duplicates. A timeout means unknown outcome, not failure. The API received and processed the order, but the response was lost, so the process saw an error and retried. Each retry created a new order because the API has no way to recognize a repeat. Retrying a non-idempotent operation on timeout is the root cause.
Design changes:
- Treat timeout as 'unknown,' not 'failed.' On timeout, do not blindly retry a create. Instead, query the API (if it offers a lookup by order reference) to determine whether the order exists before retrying. Only retry when you know it did not land.
- Send a client-generated unique reference (e.g., process instance ID or a GUID stored on the order record) with every call. Even without vendor idempotency, this lets you reconcile - look up by reference to detect existing orders.
- Adopt the vendor's idempotency key as soon as available, and send a stable key per order. This is the durable fix; make it a requirement, not optional.
- Make the step recoverable, not fire-and-forget. Record the outbound attempt (reference, timestamp, status) so a failed/unknown call can be reconciled rather than re-fired.
- Bound retries and escalate unknowns. Retry only on clear, safe-to-retry errors (e.g., connection refused before send); route timeouts to a human/reconciliation queue.
Priority: get the idempotency key, but until then, stop retrying on timeout and add reference-based reconciliation.
2. What observability and reconciliation controls should be added so a repeat incident is caught in minutes, not after 240 shipments?
Read the worked answer
The 240-shipment gap means there was no detection between the duplicate creation and the downstream consequence. Add:
- An outbound-call log per order: order reference, idempotency/reference key, request time, response status, outcome (success / known-failure / unknown). This replaces the current binary 'step completed or errored.'
- A reconciliation job that periodically compares orders marked 'sent' against the external system (by reference key) and flags orders with unknown outcomes or mismatches.
- Duplicate detection: a check for multiple external orders sharing the same reference key, alerting immediately rather than after shipping.
- Alerting on retry and timeout rates - a spike in timeouts or retries is the early signal; surface it to operations, not just logs.
- A quarantine/hold on 'unknown' outcomes: orders whose external status is unconfirmed should be held from shipment until reconciled, rather than flowing downstream.
- A dashboard of outstanding unknowns and reconciliation lag, with an owner.
Net: instrument every outbound attempt, reconcile against the external system on a schedule, hold unconfirmed orders, and alert on timeout/retry spikes. Detection in minutes requires the unknown state to be visible and actioned, not silently retried.