Retry Storm: Idempotency in an Invoice-Posting Bot
An unattended bot posts invoices from a queue into a finance ERP via a web form. The ERP is slow during month-end, and the bot's HTTP/form submission sometimes times out after the record has actually been saved. The bot currently retries any failed step up to 5 times with a 2-second fixed delay. Last month-end, 340 invoices were posted twice, requiring a manual reversal. The queue item is only marked complete after the ERP confirms with a success banner. The business wants 'zero duplicates' and 'no lost invoices' simultaneously, and has rejected any change to the ERP itself.
1. Explain why the current retry design produces duplicates, and what architectural property the bot is missing.
Read the worked answer
Why duplicates happen: A timeout is ambiguous - it does not tell you whether the write succeeded. When the form submission times out after the ERP committed the record, the bot treats it as failure and retries, creating a second record. Fixed 2-second retries during a slow month-end make this worse: the ERP is congested, timeouts spike, and the bot hammers it with retries that add load.
Missing property: Idempotency. The posting operation must be safe to repeat. Because the ERP cannot change, the bot must enforce idempotency around it:
- Idempotency key: Derive a deterministic key per invoice (e.g., invoice number + vendor + amount hash) and check-before-post: query the ERP for an existing record with that key before submitting. On retry, the lookup finds the prior success and the bot marks the item complete instead of re-posting.
- Reconciliation pass: After a timeout, do not blind-retry. Re-query the ERP to determine actual state, then act.
- Backoff with jitter: Replace the fixed 2-second delay with exponential backoff plus jitter to relieve congestion and avoid synchronized retries.
2. Design the error-handling flow so the bot can satisfy both 'zero duplicates' and 'no lost invoices' given the constraints.
Read the worked answer
Treat each queue item as a small state machine with an explicit reconciliation step. Sketch:
- Pre-check: Look up the idempotency key in the ERP. If found → mark item complete (already posted), log as reconciled. If not found → proceed to post.
- Post with bounded retries: Attempt submission. On explicit rejection (validation error), fail fast - do not retry; route to a business exception queue with the error detail.
- On timeout/unknown outcome: Do not retry the post directly. Enter reconciliation: re-query by idempotency key.
- Found → success; mark complete.
- Not found after a short wait → safe to retry the post (bounded, exponential backoff + jitter).
- Still unknown after N reconciliation cycles → move to a manual review queue with full context (key, payload, timestamps). This is the escape hatch that preserves 'no lost invoices' without risking duplicates.
- Terminal handling: Every item ends in exactly one state: completed, business-exception, or manual-review. Nothing is silently dropped.
Why this satisfies both goals: Duplicates are prevented because a post only repeats after a confirmed absence. Loss is prevented because unknown outcomes are escalated, never discarded. The manual queue is the safety valve when the ERP gives no trustworthy signal.
Also: Add a daily reconciliation report comparing bot-posted keys against ERP records to catch any drift and give the business an audit trail.