Order events on IBM MQ: per-customer ordering versus throughput
A retail company sends order-status events to IBM MQ queue ORDER.EVENTS as persistent messages. Peak volume is 12,000 events/min, average 4,000/min. Each event is about 8 KB JSON and carries customerId, orderId, eventVersion, and status. A downstream REST service PATCH /orders/{id} has p99 latency 120 ms, is idempotent by orderId for 24 hours, but returns HTTP 503 during a three-minute maintenance window every night. The business requires per-customer event ordering, no duplicate status effects, and recovery within 10 minutes after the window. Today six integration servers each run eight additional instances against the same queue. Throughput is good, but customers occasionally see statuses move backwards, and after the last maintenance window 2,300 messages sit in the backout queue with backout threshold 5. The backout queue is drained manually by an operator, then replayed in arrival order, which sometimes makes the backward-status problem worse.
1. Explain why per-customer ordering is violated and propose a message-flow topology that meets both the ordering requirement and the peak throughput. Show the capacity reasoning.
Read the worked answer
IBM MQ preserves message order on a queue only relative to a single consumer and a single in-flight message. With 48 additional instances plus multiple threads reading ORDER.EVENTS, several messages for the same customer can be processed concurrently and complete out of order. MQGET order does not help once two threads hold messages for the same customer.
The fix is to partition by customerId rather than add threads to one queue. Hash customerId into, say, 32 queues (ORDER.EVENTS.00 to .31). Each partition is consumed by exactly one instance with one in-flight message at a time. Peak is 12,000/min, or 200/s. At 120 ms downstream latency, one serial consumer can sustain about 8.3/s, so 32 partitions provide roughly 266/s of capacity before headroom is lost. That meets 200/s with room for retries and maintenance catch-up.
Do not run multiple threads on the same partition queue, even if the downstream is idempotent; idempotency prevents duplicate effects, not reordering. Also carry eventVersion and reject or ignore an event whose version is lower than the last committed version for that order. The topology change should be rolled out by draining one partition at a time, not by starting 48 consumers on the old queue.
2. Design the failure handling for the nightly three-minute HTTP 503 window. The operator wants to let messages roll back and go to the backout queue, then replay them afterward. Evaluate that approach and propose a better one.
Read the worked answer
Rolling back every 503 is the wrong tool here. A rollback releases the message and lets the consumer take the next one, so later events for the same customer can pass the failed event. After backout threshold 5, the event is removed from the main queue entirely, which breaks the partition's ordered stream. The manual replay then interleaves partitions and makes backward statuses more likely.
Instead, treat 503 as a planned, retryable outage. On the partition consumer, set a bounded HTTPRequest timeout and detect 503 or connection failure. Pause that partition, retry the same message with exponential backoff and jitter, and do not take the next message until it succeeds or a hard deadline expires. A 10-second timeout with retries inside the three-minute window is acceptable if the partition has enough throughput headroom to catch up afterwards. If the deadline expires, park the partition's message on a dedicated retry queue with its sequence metadata and alarm; do not release later messages.
For the 2,300 already-backed-out messages, stop intake, wait for the health check to pass, then replay by partition in original sequence order. Use eventVersion to discard stale replays. Raise the backout threshold and reserve the backout queue for poison messages, not planned maintenance.
3. A colleague proposes raising each integration server from eight to sixteen additional instances to clear the backlog faster. What is your judgement?
Read the worked answer
Reject it as the primary remedy. Adding instances to the same queue increases concurrency across the same customer's messages, so it makes the ordering symptom worse. It also increases the number of simultaneous HTTP calls during the recovery window, which can overload the downstream service as it comes back and trigger a retry storm.
The backlog is a partition-level recovery problem, not a raw thread-count problem. If 32 partitions cannot catch up within ten minutes, add partitions first, for example 48 or 64, and rebalance customer hashes. If the bottleneck is the downstream service, more ACE threads only create more waiting. Keep one in-flight message per partition, measure per-partition lag, and scale partitions only when the downstream can absorb the extra concurrency.
If additional instances are used at all, they should own different partition queues, not share one, and the change should be capacity-tested with the maintenance window simulated. The decision criterion is per-customer ordering first, then end-to-end catch-up rate, not queue depth alone.