Queue Depth Growth and Oversized Messages on IBM MQ Order Capture

Arithmetic first. Steady state is fine: 3,000 rps peak production against 5 consumers times 120 rps equals 600 rps of consumption. During a flash sale the backlog grows by roughly 2,400 messages per second, so a 500,000-deep queue fills in about three and a half minutes of sustained peak. Raising MAXDEPTH to 2,000,000 buys roughly 14 minutes instead of 3.5 - it converts a fast, visible failure into a slower one and lengthens the recovery tail. Meanwhile disk is at 78%: 240,000 messages at 40 KB is about 9.6 GB of payload, and 2,000,000 messages at the same size is about 80 GB before accounting for persistent-message log writes, index overhead and copy-on-write behaviour during restart. You would almost certainly hit a full filesystem, which is a far worse failure than a full queue because it can stop the queue manager and affect unrelated queues.

What a full queue actually does for you. It is backpressure: producers get an explicit failure they can retry or shed, and the system degrades in a controlled way. Silently absorbing 2,000,000 messages instead means orders sit in the queue for an hour and the business discovers the delay at the customer, not the producer.

Instead: match consumption to production during peaks (add consumers, raise consumer concurrency, remove the 120 rps ceiling if it is client-side), shorten the message (40 KB average suggests payload you could move to object storage with a claim-check reference), and apply backpressure deliberately at the front end. Raise MAXDEPTH only as a modest buffer for restart windows, with disk headroom calculated and monitored.

Check every length limit on the path, not just the queue. A message must fit the queue manager's maximum message length, the destination queue's MAXMSGL, any transmission queue in the path, and both ends of the channel. The usual cause of this symptom is that the queue was sized but the channel was left at a default that is smaller than the payload, so the put succeeds locally and the message is rejected on the way across. Verify each of these explicitly rather than only looking at ORDERS.REQUEST.

Then decide whether 6 MB messages should exist. MQ will carry them, but you pay for it everywhere: log writes for persistent messages scale with message size, channel batch sizes and in-doubt windows get coarser, memory requirements for consumers rise, and every future queue and channel in the path must be re-sized and remembered. Large binary attachments are a poor fit for a messaging backbone whose job is to move small order events reliably.

Recommended pattern: put the attachment in object storage, publish its location and checksum in the order message, and let the consumer fetch it. Messages drop back to well under 100 KB, the queue depth arithmetic above improves, and the supplier contract becomes a URL rather than a payload. This also removes the need to raise limits across the entire estate.

If you must carry them: raise MAXMSGL explicitly on the queue, transmission queue, both channel ends and the queue manager, test a restart with the largest message present, and confirm the log filesystem has headroom for the increased write volume.

The risk is a silent data loss channel. A backout queue is not a graveyard; it is work that has been removed from the main flow and is invisible to normal monitoring. An undrained backout queue means orders accumulated there are never processed and never reported, and because the queue is empty the team may conclude nothing is failing. Worse, if the backout queue itself has a backout threshold configured, messages can traverse to the dead letter queue or, depending on put settings, be discarded - losing the order entirely.

What caused the rollbacks here. The transaction failures coincided with peak queue depth and inventory-service contention, so most backouts are probably not poison messages at all: they are transient dependency failures that exhausted a three-attempt budget during a brownout. That matters, because a retry budget designed for corrupt payloads should not be the same budget used for a slow downstream service.

Manage it as three channels:

1. Transient failures should be retried with backoff and jitter, not immediately re-queued to the back of the queue three times. Re-queued messages compete with fresh traffic and deepen the backlog that caused the failure.
2. Poison messages should reach the backout queue quickly and alert. Distinguish them by validating payload shape before the downstream call.
3. The backout queue itself needs a monitored depth alarm, a defined owner, a browsing tool, and a documented replay procedure that preserves the original message identity so consumers can deduplicate.

Second-order issue: the backout queue indicates that ordering is already being broken - five competing consumers plus re-queued retries cannot preserve per-order sequence. If order matters, that requires a different consumption design, not a larger retry count.