The process that called the warehouse twice

Separate the wait from the process. An automatic activity that blocks on a slow HTTP call couples process state to a third party's availability, and every instance that errors must later be repaired by hand - which is exactly what produced the 240 duplicates. Move the outbound call into a queue: the process writes a shipment request row and moves on; a dedicated worker (timer-driven) sends it, records the outcome, and the process is driven forward by that outcome.

Make the call idempotent. Generate a deterministic request key (order id plus attempt purpose) and persist it before calling. Before re-sending, query the warehouse by that key to see whether it already accepted the request. Now Resume, retry, or a duplicated queue row cannot produce a second shipment. Without this, no retry policy is safe.

Handle failure explicitly. Wrap the call so a timeout or 5xx moves the request to a retry state with backoff and a bounded attempt count, then to manual review; do not let an unhandled exception park 1,900 process instances in error. A three-hour outage should then cost throughput, not correctness.

Change the resume procedure. Resume is a bulk action applied to instances that failed for unknown individual reasons. The playbook should require idempotency verification first, which is precisely what was missing when 240 orders shipped twice.

The design problem is that each run tries to do everything. A five-minute schedule cannot absorb a scan of 200,000 rows, so runs never catch up and the backlog compounds.

Changes:

• Bounded batches instead of a full sweep. Process N rows per iteration (say 500) in a loop with an overall execution cap, and checkpoint the last processed id so the next run continues rather than restarts. Throughput becomes predictable and a bad run cannot hold the database for minutes.
• Index the filter. Contains on a notes column means a function on the column and a scan every five minutes. Replace with a status flag or a dedicated work-queue table populated at write time, and index it, so each batch touches only relevant rows.
• Delete in set operations, not row by row. One statement per batch instead of N round trips.
• Keep the execution well inside the interval. Set the timer's Timeout below the schedule period so a stuck run is cut off instead of silently colliding with the next one, and guard the entry point with a lock flag so two executions can never work the same rows.
• Alert on duration and backlog size, not only on errors, because this timer was failing visibly in Service Center for days without anyone acting.

Verify with numbers: rows processed per run, run duration, and backlog trend. If the backlog still grows, the answer is more frequent bounded runs or a queue, not a longer scan.

A human activity timeout is an escalation, not a cancellation. When the four hours elapse, the process leaves the human activity through the timeout connector, but the real-world pick still happens. If the timeout path also produces a shipment - or if the late confirmation is later accepted as a new pick - the same physical pick is recorded twice. The timeout changed the process's belief about the world without changing the world.

What the timeout path should do:

• mark the order as Escalated and notify the warehouse supervisor, so the exception has an owner;
• not create a shipment; it should only open a reconciliation task;
• leave the order in a state where a late confirmation is still valid but checked, so the confirmation action accepts a pick only when the order is AwaitingPick or Escalated, and records the state transition rather than blindly writing a shipment;
• be idempotent itself, so a supervisor clicking twice or a resumed instance cannot add a second shipment.

The general principle to state in the review: every path out of a human activity, including timeout and error, must leave the process in a state that a compensating action can reason about. Timeouts are common at peak volume - four hours is short for a night shift - so this path is exercised routinely, not exceptionally. Treat it as a normal branch and test it as one.