Dead-Letter Queue Fills After a Queue Is Deleted
At 07:00 the operator finds 2,100 messages on SYSTEM.DEAD.LETTER.QUEUE on CLAIMS.QM. Overnight, a storage cleanup removed the local queue CLAIMS.IN on QM2; the sender channel from QM1 to QM2 stayed RUNNING. QM2's channel agent put each undeliverable message to its dead-letter queue. The application team wants the messages replayed before the 09:00 claims cut-off. The queue CLAIMS.IN has now been recreated with the same attributes, but no messages have been moved. The DLQ is not full. A junior operator suggests 'just purge the DLQ and ask the sender to resend'; the sender has already deleted its copies. You need to inspect the messages, restore them safely, and prevent a repeat without disabling dead-letter handling. The messages are persistent, and the DLQ handler is not running. The queue manager is otherwise healthy.
1. How do you confirm the cause without destroying the evidence, and why is purging the DLQ the wrong first move?
Read the worked answer
Do not purge. First inspect the DLQ header, because it records the original destination and the failure reason. Use amqsbcg SYSTEM.DEAD.LETTER.QUEUE CLAIMS.QM (or a browse client) and look at the MQDLH fields DestQName, DestQMgrName, and Reason. A reason of 2085 (unknown object name) with DestQName(CLAIMS.IN) and DestQMgrName(QM2) confirms that QM2's MCA could not put to the recreated queue at the time. Check DIS QL(SYSTEM.DEAD.LETTER.QUEUE) CURDEPTH and DIS CHSTATUS(TO.QM2) to confirm the channel is running. Also check whether the messages are persistent; if they are, replay is appropriate. The fact that the DLQ is not full means there is time to do this properly.
2. What is a safe replay procedure now that CLAIMS.IN has been recreated?
Read the worked answer
After recreating CLAIMS.IN, verify its MAXMSGL, DEFPSIST, and authority match the sender's expectations. Replay in a controlled way: stop the sending channel or put the sender's queue in a state where it cannot add more messages, then use the dead-letter queue handler with a rule that matches the reason and retries to the original destination, for example REASON(MQRC_UNKNOWN_OBJECT_NAME) ACTION(RETRY). Run it first against a copy of the DLQ or with a small batch, and confirm the target queue depth rises. If runmqdlq cannot resolve the destination, write a small MQ program that reads the DLH, opens DestQName on DestQMgrName, and puts the payload with the original message ID preserved. The consumer must tolerate duplicates, because replay and the original send can overlap.
3. What monitoring and process changes prevent a repeat?
Read the worked answer
Run a DLQ handler continuously rather than only during incidents, and alert on CURDEPTH greater than zero for more than a few minutes. Change the storage cleanup procedure: before deleting a queue that is a channel or cluster destination, stop or quiesce the relevant channel, remove the queue from the cluster, and check for in-flight messages. Add a pre-delete check for CURDEPTH and open handles. Do not disable dead-letter handling; the DLQ is the safety net that made this recovery possible. Also record the reason code trend so a repeat is caught at 10 messages, not 2,100.