Mobile checkout: synchronous REST endpoint with slow downstream systems
A retail chain’s mobile checkout calls a CAI REST service to submit orders. The mobile SDK aborts any call that exceeds 5 seconds and retries once with the same client-generated order ID. The orchestration must validate the cart, reserve inventory in an on-premises ERP (p95 18 seconds), authorize payment through a cloud gateway (p95 9 seconds), and publish an order-confirmed event. At peak the service receives 120 requests per minute. The current synchronous process occasionally runs 30 seconds, so the mobile app retries and creates duplicate orders; finance has also found payment authorizations with no corresponding reservation. Operations wants a response within 2 seconds, no duplicate side effects, and a way for support to query order status. The ERP API is idempotent only when given a reservation key; the payment gateway supports an idempotency header but holds authorizations for 7 days.
1. Recommend the invocation and response pattern for the checkout endpoint. Explain why the current synchronous design fails and what the client should receive.
Read the worked answer
The synchronous design fails because the caller’s 5-second deadline is shorter than the p95 of the downstream work: 18 seconds for ERP, 9 seconds for payment, plus overhead. Holding the HTTP request open does not make the work faster; it converts a slow success into a client timeout and a retry.
Return 202 Accepted with a correlation ID and a status URL after minimal validation and durable acceptance. The request thread should only validate the payload, enforce an idempotency key, persist an order-request record, and enqueue work for an asynchronous orchestration process, such as a JMS- or event-triggered process. The orchestration then calls ERP and payment. Support queries the status endpoint by correlation ID. If the business truly needs a synchronous answer, it must either shorten the downstream SLA or accept a longer client timeout; neither is realistic here. A 2-second response with a 202 and eventual status is the consistent contract. Do not fake a 200 before inventory and payment have completed.
2. How should the process prevent duplicate orders and duplicate payment authorizations when the mobile app retries? Include the key design elements and where state is stored.
Read the worked answer
Use the client order ID as the idempotency key, but do not trust it alone: store a hash of the cart and customer with it, because a buggy or malicious client could reuse an ID for a different cart. Create a unique constraint in a durable store, such as a database table or a process object with a unique key. On the first request, insert RECEIVED and return the correlation ID. On a retry, the insert conflicts; read the existing record and return the same correlation ID and status instead of starting a second orchestration.
Pass a derived reservation key to ERP and an idempotency header to the payment gateway, so even an at-least-once orchestration cannot create two reservations or two authorizations. Persist each state transition (RECEIVED, RESERVED, AUTHORIZED, CONFIRMED, COMPENSATED) with timestamps. Keep the idempotency record for at least the payment authorization hold period plus a margin, for example 8 days, not just the HTTP retry window.
3. Inventory reservation and payment authorization both take time. Should they run in parallel? Describe timeout, retry, and compensation rules that keep the process consistent.
Read the worked answer
Parallelism is acceptable only because payment authorization can be voided and inventory reservation can be released. Run inventory and payment concurrently to reduce elapsed time, but give each a timeout below the orchestration budget, for example 25 seconds for ERP and 15 seconds for payment, and record which side succeeded. If inventory fails after payment authorized, void the authorization using the same idempotency key; if payment fails after inventory reserved, release the reservation.
Retry only idempotent calls or calls protected by a stable idempotency key; do not retry a payment authorization with a new key. Use bounded retries with backoff for transient network errors, but fail fast on business declines. The compensation steps must themselves be idempotent and recorded. If the process crashes between the two calls, a recovery job must reconcile RESERVED without AUTHORIZED and AUTHORIZED without RESERVED using persisted state, not memory.