ERP Integration Sync Failures Under Partial Outages
Adobe Commerce integrates with an ERP for inventory and order export. Inventory is pulled via a nightly REST batch (~120k SKUs) and order export is pushed via a synchronous REST call at order placement. The ERP has frequent 30–90s outages. Symptoms: (1) orders fail to place when ERP is down (checkout blocks on the sync call), (2) nightly inventory sync sometimes times out mid-batch, leaving half the SKUs updated, (3) duplicate orders appear in the ERP after retries. The integration code retries 3x immediately on failure.
1. Explain why checkout blocks on the ERP call and why immediate retries cause duplicates.
Read the worked answer
Checkout blocks because order export is a synchronous REST call inside the order placement flow - the customer's request thread waits for the ERP. Any ERP outage (30–90s) exceeds the request timeout, so the order fails. This couples customer-facing latency and availability to a flaky third party; unacceptable.
Duplicates: the integration retries 3x immediately. If the ERP actually received the order but the response was lost (timeout after commit), the retry sends it again - the ERP has no idempotency key, so it creates a second order. Immediate retries also amplify load during an outage (retry storm), worsening the 30–90s window.
Fix direction: make order export asynchronous (message queue) and idempotent (send a stable idempotency key = order increment ID or a UUID stored on the order). Retries must be delayed/backed off and only for retryable errors.
2. Design the order export integration to be resilient and exactly-once. Specify the queue, idempotency, and failure handling.
Read the worked answer
Architecture:
- On order placement, publish an
order.exportmessage to a message queue (RabbitMQ) containing order ID + idempotency key (use the orderincrement_idor a generated UUID persisted on the order). Checkout returns immediately; no ERP call on the request path. - A consumer processes the message and calls the ERP with the idempotency key in a header. The ERP must dedupe on that key (contract requirement).
- Retry policy: exponential backoff with jitter (e.g., 1m, 5m, 30m, 2h), max attempts, then route to a dead-letter queue with alerting. Never retry immediately 3x.
- Idempotency on Commerce side: store export status per order (
export_status,idempotency_key) so a redelivered message is a no-op if already exported. - Outbox pattern if you need guaranteed publish: write an outbox row in the order transaction, a relay publishes it. This closes the gap where the order commits but the message publish fails.
This makes export exactly-once (effectively) and decouples checkout availability from the ERP.
3. The nightly inventory batch times out mid-run. Redesign it for partial-failure safety and correctness.
Read the worked answer
Problem: a single 120k-SKU batch that times out leaves half-updated state with no way to know which half succeeded.
Redesign:
- Chunk the batch into pages (e.g., 1,000 SKUs) and process each chunk as a unit; commit per chunk so a failure doesn't roll back everything.
- Checkpointing: record last processed offset/page in a control table; resume from checkpoint on failure instead of restarting.
- Idempotent upserts: inventory updates are naturally idempotent (set qty to value), so re-processing a chunk is safe.
- Per-chunk error isolation: a bad SKU shouldn't fail the whole chunk; log and continue, aggregate failures for review.
- Freshness marker: write a
last_synced_atper SKU so reporting/staleness alerts know which SKUs are current. - Backpressure/timeouts: set per-chunk timeouts and a total run budget; alert if the run exceeds budget.
Also move to a pull-with-pagination API or file drop + consumer rather than one giant request. Verify with a simulated mid-run failure that the system resumes cleanly and no SKU is left with a torn value.