The Integration That Retries Forever
An integration calls a third-party payment API to charge a card and write back the result to a Payment record. The API is occasionally slow and returns HTTP 500 under load. The current integration has a 60-second timeout and an automatic retry on failure (up to 5 attempts). Last week, a customer was charged twice: the first call succeeded on the vendor side but timed out on Appian's side, and the retry charged again. Separately, the team reports that during vendor incidents, process instances pile up and eventually error out with no record of what happened. The vendor supports an idempotency key header.
1. Diagnose the double charge and specify the fix.
Read the worked answer
This is the classic at-least-once delivery problem, and it is not solved by tuning the timeout.
What happened: Appian sent the charge, the vendor processed it, but the response did not arrive within 60 s. Appian saw a timeout, treated it as a failure, and retried. The retry was a new charge because nothing told the vendor it was the same request.
Fix - idempotency:
- Generate a stable idempotency key per logical payment attempt (e.g. derived from the
Paymentrecord ID, not a random value per HTTP call) and send it in the vendor's idempotency header on every attempt, including retries. - The vendor then returns the original result for a repeated key instead of charging again. This is the only correct fix; timeouts and retry counts cannot distinguish "failed" from "succeeded but I didn't hear back."
- Persist the key and the attempt state on the
Paymentrecord before the call, so a retry after a process restart reuses the same key rather than minting a new one.
Also: treat a timeout as unknown, not failed. The correct next step for an unknown outcome is to query the vendor for the status of that idempotency key, not to blindly re-charge. Blind retry on timeout is the specific anti-pattern here.
2. How should retries, timeouts, and failure visibility be designed so vendor incidents are recoverable?
Read the worked answer
Retries. Retry only on errors that are safe to retry - 5xx and timeouts with an idempotency key; never retry 4xx (bad request, declined card) because the answer will not change. Use exponential backoff with jitter, not 5 immediate attempts; hammering a struggling vendor deepens the incident. Cap total attempts and total elapsed time.
Timeouts. 60 s is long for a payment call. Set a timeout appropriate to the vendor's p99 (e.g. 15–20 s) so you fail fast into a known state rather than holding threads. A shorter timeout plus idempotency is safer than a long timeout without it.
Failure visibility. The current design loses information. Instead:
- Write a status on the
Paymentrecord (Pending / Succeeded / Failed / Unknown) and an integration log entry per attempt with timestamp, response code, and correlation/idempotency key. - Route exhausted-retry cases to an exception queue (a record type + a task) for human review, rather than letting process instances error out invisibly. A human can then query the vendor and resolve the Unknown state.
- Separate "the vendor is down" from "this payment is bad": the former is an operational alert, the latter is a business exception. They need different handling and different owners.
The principle: integrations must be designed for partial failure and unknown outcomes, with durable state written before the call so that recovery does not depend on the process instance surviving.