Real-Time Order Integration Proposed Without Capacity or Idempotency Controls
An online retailer runs order capture on a cloud commerce platform and fulfillment on an on-premises ERP. Orders are sent by nightly batch; the average delay is 7 hours, and peak-season delay reached 11 hours. Customer service sees 'order not found' for up to 9 hours, generating 1,200 calls a day in December. The integration team proposes replacing batch with real-time API calls for every order event. The ERP can handle 40 API calls per second; peak order rate is 85 per second. The network link has 99.5% availability, and the ERP has maintenance windows every Sunday. The CFO is concerned about duplicate orders if the API retries; the current batch has a reconciliation report that matches 99.98% of orders. The team has budget for either an event broker or an ERP upgrade, not both. The business wants sub-15-minute visibility for orders over $500 and can accept hourly for the rest.
1. Evaluate the proposal to replace batch with real-time API calls for every order event, and recommend an alternative.
Read the worked answer
The proposal fails on capacity, reliability, and cost. Peak order rate is 85 events per second against an ERP ceiling of 40, so synchronous calls would create a backlog or reject orders at exactly the worst time. A 99.5% network link and Sunday ERP maintenance mean the API path cannot be the sole delivery mechanism. Calling the ERP synchronously for every event also couples order capture to fulfillment availability.
Recommended alternative: a hybrid, event-driven design.
- Insert an event broker between commerce and ERP. The broker buffers bursts, decouples availability, and supports retry and replay. This is the budget choice over an ERP upgrade.
- Prioritize by business value: route orders over $500 through a low-latency event stream for sub-15-minute visibility. Batch or micro-batch the remainder hourly, which meets the stated business requirement.
- Throttle and shape traffic to the ERP at or below 40 calls per second, with queue depth monitoring and backpressure.
- Preserve reconciliation across both paths so Finance can match orders and revenue.
This meets the two service tiers without pretending the ERP can absorb every event synchronously. It also uses the event broker to solve the maintenance-window problem through store-and-forward rather than manual catch-up.
2. How should the integration team address the CFO's duplicate-order risk?
Read the worked answer
Do not promise exactly-once delivery across a network boundary; design for idempotency and reconciliation instead.
Controls:
- Idempotency key per order event, generated by the commerce platform and carried through retries. The ERP must reject or ignore a repeated key rather than create a second order.
- At-least-once delivery with deduplication at the ERP boundary. Document that duplicates are possible but controlled, and test the retry path with injected failures.
- Sequence numbers and event timestamps so out-of-order or late events can be detected and reconciled.
- Business reconciliation: retain the 99.98% batch reconciliation control and extend it to the event path. Compare order counts and amounts by day and by channel; investigate breaks within one business day.
- Compensating transactions for confirmed duplicates, with an owner and SLA, rather than assuming the problem will not occur.
The CFO's concern is legitimate because an API retry without idempotency can double-ship or double-bill. The answer is an engineered control set, not a verbal guarantee. The reconciliation report is a strength of the current process and should survive the redesign.
3. What operating model should accompany the new integration design?
Read the worked answer
The operating model must cover monitoring, failure handling, and business communication:
- End-to-end monitoring: event lag, queue depth, ERP call rate, dead-letter count, and order visibility time, segmented by high-value and standard orders. Alert thresholds should align to the 15-minute and hourly commitments.
- Failure handling runbook: automatic retry with backoff, dead-letter queue review every 30 minutes during peak, replay procedure after ERP maintenance, and network-outage buffering. Name an on-call owner for integration and a business owner for order exceptions.
- Maintenance-window protocol: drain the broker before Sunday maintenance, confirm ERP availability, and replay with idempotency keys after the window. Publish order-visibility expectations to customer service.
- Daily reconciliation and weekly service review: Finance, Operations, and IT review breaks, duplicate attempts, and near misses. Track 99.98% match as a service level, not an informal report.
- Change control: any new event type must define idempotency, schema, volume estimate, and reconciliation rule before go-live.
This prevents the new architecture from trading a visible batch delay for invisible queue failures. The customer-service call volume is a business symptom; the operating model should report it alongside technical metrics.