The Integration That Retried Into a Duplicate Storm
A process calls a REST integration to create invoices in a finance system. The endpoint occasionally returns a 502 during peak load. The integration is configured to retry on error up to 5 times with no delay. During an incident, the finance system processed the same invoice 4 times because the first call actually succeeded server-side but the response was lost. Finance now sees duplicate invoices and has frozen the integration. The process currently generates the invoice number locally and sends it in the request body; the finance system does not deduplicate. The team wants to know how to make the integration safe to retry.
1. Explain why retries produced duplicates and what property the integration is missing.
Read the worked answer
The calls are not idempotent. A 502 means the client did not receive a response; it does not mean the server did not process the request. With immediate retries and no deduplication key, each retry created a new invoice.
The missing property is a stable idempotency key that the finance system honors: the client sends the same key on every retry, and the server returns the original result instead of creating a new record. The current design sends a locally generated invoice number, but the finance system does not deduplicate on it, so it provides no protection.
Fixes:
- Agree with finance on an idempotency key (e.g., a process-generated GUID stored on the source record) and have the endpoint enforce uniqueness on it.
- Generate the key before the first call and persist it, so retries reuse it.
- Add exponential backoff with jitter instead of five immediate retries; immediate retries amplify load during an incident.
2. The finance team cannot change the endpoint this quarter. What can the Appian side do now to prevent duplicates?
Read the worked answer
Without server-side deduplication, the client must make retries safe by checking before creating:
- Before each retry, call a read endpoint (or a query) to see whether the invoice with this key already exists. If it does, treat it as success and stop.
- Persist the request key and a status (
sent,confirmed,unknown) on the source record. Only retry when status isunknown. - Cap retries and route exhausted cases to a reconciliation queue for a human, rather than looping indefinitely.
- Add backoff with jitter and a circuit breaker so peak-load 502s do not trigger a retry storm.
This is a compensating control, not a fix: it narrows the duplicate window but cannot close it entirely if the read is eventually consistent. Document that limitation and prioritize the idempotency key next quarter. Until then, consider a manual approval gate on high-value invoices.