Boomi Process Consumes a Messaging Queue Without Idempotency
A manufacturer uses a Boomi process to consume order events from a message queue and create orders in an ERP. The queue guarantees at-least-once delivery. During a broker failover last month, roughly 1,800 events were delivered twice within a few minutes, and the ERP created duplicate orders; finance spent two days reversing them. The process currently maps the event payload directly to an ERP create operation and has no record of what it has already processed. Volume is about 50,000 events per day with bursts to 400 per minute. The ERP has no native duplicate detection on the order number because the order number is assigned by the ERP, not the source.
1. Explain why at-least-once delivery makes duplicates inevitable here and design a deduplication approach that works given that the ERP assigns the order number.
Read the worked answer
At-least-once delivery means the broker guarantees an event is delivered at least once, never exactly once. On failover, an event whose acknowledgement was not confirmed is redelivered. The consumer must therefore be idempotent; the broker will not do it. This process is not idempotent because it performs an unconditional create, so every redelivery becomes a new ERP order. The ERP's lack of duplicate detection on order number removes the last safety net - and since the ERP assigns the number, the consumer cannot use it as a dedup key anyway.
The dedup key must come from the source. Use a stable event identifier or a deterministic composite of source order reference plus event version, and persist it in a durable store the process checks before calling the ERP. The check-then-create sequence has a race under concurrency, so the store must enforce uniqueness atomically - insert the key first and proceed only if the insert succeeded, or use a conditional write. A simple read-then-write check will still admit duplicates at 400 events per minute across concurrent executions.
Record the ERP-assigned order number back against the dedup key after creation, so retries and support investigations can correlate. Set a retention period longer than any plausible redelivery window.
2. The team proposes storing dedup keys in process memory to avoid the extra latency of an external store. Evaluate that proposal and state what you would do instead.
Read the worked answer
Reject it. In-memory state is scoped to a single process execution or, at best, a single runtime instance, and it is lost on restart - which is exactly when redeliveries happen. The failover scenario that caused the incident would defeat it entirely, because the instance holding the keys is the one that failed. It also does not work across multiple runtime instances processing the queue in parallel, since each has its own memory and would not see the others' keys.
The latency concern is real but overstated. A keyed lookup against a durable store is typically single-digit milliseconds, against a 400/minute burst that is trivial load. If latency is genuinely tight, the answer is a store designed for it - a fast key-value or indexed table with a unique constraint - not abandoning durability.
What I would do: an external durable dedup store with an atomic unique insert, checked before the ERP call, with the ERP order number written back after success, and a retention window comfortably longer than the broker's redelivery horizon. Then load-test at burst rate to confirm the store is not the bottleneck.