A Fulfillment Pipeline That Loses State at the Worst Moment
Order fulfillment is one Lambda function invoked synchronously by API Gateway. It calls reserveInventory, chargePayment (up to 8 seconds), then createShipment (up to 45 seconds), and finally polls a DynamoDB flag waiting for a warehouse operator to approve large orders - an approval that can take up to three days. Payments are charged before the approval wait begins. Last month 11 customers were charged twice: when createShipment failed and the handler returned HTTP 500, the API client's own retry logic resubmitted the whole order, charging payment again. When an approval is slow, the handler dies at the 15-minute Lambda timeout and the client retries produce the same effect. Inventory reservations are never released on failure, so the system also leaks capacity.
1. What is structurally wrong with this design, and why does each of the reported failures follow from it?
Read the worked answer
The orchestration state lives only in a running process, and that process cannot live long enough. Several distinct problems compound:
- Lambda's 15-minute maximum is shorter than the work. A 3-day human approval cannot be awaited by any compute process. Polling a DynamoDB flag for three days from a Lambda is not possible, and paying for a sleeping execution is not possible either.
- A synchronous chain makes the client the retry engine. The API client's retry is retrying the entire order, including the already-successful payment charge. That is the duplicate charge: exactly-once fails not in DynamoDB but at the API boundary.
- No compensation. When step 3 fails, steps 1 and 2 stay committed. There is no rollback path, hence the leaked inventory reservations.
- No idempotency key. Nothing on the request identifies it as a duplicate of an order already in progress.
The common root is that a single short-lived, non-durable, synchronous function is being used as a workflow engine. It has no memory across invocations and no way to express "this step succeeded, that one failed, here is the compensating action."
2. Describe the replacement architecture and how it specifically prevents the double charge and the inventory leak.
Read the worked answer
Move orchestration into a Step Functions Standard workflow, and make the API asynchronous.
Flow: API Gateway → startOrder Lambda → StartExecution → returns HTTP 202 with the execution ARN. The client polls a status endpoint or receives a webhook; it is never allowed to retry the business transaction synchronously.
Idempotency: use the order ID as the execution name. Step Functions rejects a duplicate StartExecution with ExecutionAlreadyExists, so a client retry is a no-op rather than a second charge. Pair that with an idempotency key on the payment gateway call itself - two layers, because the payment provider is the only party that can truly guarantee it.
Human approval: use the task token callback pattern. A waitForTaskToken state emits a token, the internal console stores it and later calls SendTaskSuccess or SendTaskFailure. The workflow waits three days at no compute cost.
Compensation: wrap createShipment in Retry (exponential backoff, bounded attempts on transient errors only) with a Catch that transitions to compensating states - refundPayment, releaseInventory - then to a terminal failure state. Every step that has a side effect gets a matching undo.
Observability: each state is visible in the execution history, so "where is order 1234?" has an exact answer, and a stuck approval is a visible state rather than a timed-out Lambda.
3. Could this be built on an Express workflow instead? Justify the choice with the constraints in play.
Read the worked answer
No - Standard is required here. Express workflows have a maximum duration of 5 minutes, and this workflow must hold a human approval for up to 3 days. That alone disqualifies Express; the timeout is not configurable.
Two further disqualifiers:
.waitForTaskTokenis not available in Express. The callback pattern that makes the approval cheap and durable cannot be implemented, which would force you back to polling.- Express executes at-least-once. For a workflow whose failure mode is a duplicate payment charge, at-least-once semantics on the orchestration itself reintroduce exactly the bug being fixed. Standard is exactly-once for the workflow's own execution progression (side effects in downstream services still need their own idempotency).
Where Express would be right: high-volume, short-lived, event-driven flows - for example, per-request enrichment under 5 minutes at thousands of executions per second, where the price difference (Express is priced per request and GB-second rather than per state transition) matters more than durability.
Cost check on Standard: the workflow does roughly 10–15 state transitions per order. That is a real per-order cost, so keep the state machine lean - avoid a state per microstep, and do not model the poll loop as a state machine cycle; that is what the task token removes.