Duplicate Tenders After the TMS Bridge
A freight broker acquires a regional carrier and must merge two transport management systems. Shippers tender loads as EDI 204 messages and as REST calls; carriers reply with 990 acceptances and 214 status events. To hit a contractual deadline, a team deploys a bridge that forwards every inbound tender to both transport management systems. Within three weeks shippers report duplicate tenders: the same load is accepted twice in about 1 in 400 cases. Logs show the bridge retries after a 5-second timeout while the downstream system is still processing, and that system creates a new load for each POST. Separately, 214 status events arrive out of order - 'delivered' occasionally precedes 'dispatched' - because each event is routed independently through a message broker with three partitions. Volume is 22,000 tenders per day. The integration lead proposes adding a dead-letter queue and increasing the retry timeout to 30 seconds.
1. Diagnose the duplicate tender problem. Why does increasing the retry timeout not fix it, and what does?
Read the worked answer
The root cause is not timing. It is at-least-once delivery against a non-idempotent consumer. A timeout means the client did not hear back; it does not mean the server failed. The first POST often succeeds after the bridge has given up, and the retry creates a second load. A 30-second timeout reduces the frequency of that race and changes nothing about its existence - any partition, broker redelivery, or slow batch reproduces it. Worse, it lengthens the window during which two systems are processing the same tender.
What actually fixes it:
- A natural idempotency key: shipper tender ID plus version. Persist it at ingress with a unique constraint before any downstream call.
- Make create idempotent downstream - upsert keyed on tender ID, returning the existing load on repeat.
- Retries reuse the same key. The bridge writes to an outbox before dispatch so a crash cannot lose or duplicate.
- Stop fanning out to both systems. Assign each load one owning system by shipper or lane; dual delivery is the reason a single duplicate becomes a double acceptance.
A dead-letter queue is useful for visibility, but on its own it just moves the symptom.
2. Design the ordering guarantees for 214 status events, and state what the system should do when it receives 'delivered' before 'dispatched'.
Read the worked answer
The bug is the routing key. Three partitions with independent routing means events for one load can be processed concurrently and in any order. Ordering must be scoped to the entity that has a lifecycle.
Design:
- Partition key is the load ID, so all events for one load land on the same partition and are consumed by a single consumer instance.
- Each event carries a sequence number, and the consumer maintains a state machine: tendered → accepted → dispatched → delivered. Transitions must be monotonic.
- A stale or out-of-order event is not applied to current state. It is persisted, and the current state is recomputed from the ordered event history.
On 'delivered' before 'dispatched': do not reject it and do not blindly overwrite state. Buffer it against the load, apply any missing prior transition when it arrives, and if it does not arrive within a defined window, raise a reconciliation exception to operations. A missing 'dispatched' may mean a genuine gap in the carrier feed, and silently dropping the delivery would be worse than the ordering problem.
Contract point. Sequence and ordering expectations belong in the published interface contract, with consumer-driven contract tests, so the next consumer is not surprised.
3. Plan the decommissioning of the bridge with measurable exit criteria.
Read the worked answer
Sequence:
- Move to single ownership per load - route by shipper or lane, not by fan-out. This is the change that removes the duplicate class, and it should land before anything else.
- Run the bridge in shadow mode: it observes and compares, but does not forward.
- Migrate shippers in cohorts, smallest and most tolerant first, with a replay capability if a cohort goes wrong.
- Keep the old path read-only for a defined retention window before switching it off.
Exit criteria, stated numerically:
- Zero duplicate tenders for 30 consecutive days across the migrated population.
- 100% of 214 events applied in order or resolved through reconciliation within the agreed window.
- Daily tender counts reconcile between ingress, both systems, and the shipper acknowledgements, with a signed report.
Governance: version the interface contract, add contract tests to the pipeline, and record an ADR for the ownership split so the next integration does not reintroduce fan-out as a shortcut. Without numeric exit criteria, a 'temporary' bridge becomes permanent by default.