Multi-Instance Queue Manager Will Not Fail Over
MI.QM is a multi-instance queue manager on two RHEL nodes, MI-A and MI-B, with the queue manager data and logs on shared NFSv4 storage mounted at /MQHA. MI-A is active; MI-B is standby. During a patching test, MI-A was powered off. MI-B did not take over. On MI-B, dspmq -m MI.QM -x shows the instance as 'Status(Starting)' and then 'Status(Ended immediately)'. The shared mount is present on both nodes, but the mqm user on MI-B has UID 1001 while on MI-A it is UID 1002. The NFS export was recently rebuilt by the storage team. The business needs automatic failover within five minutes, and there is no maintenance window today. You must restore service on MI-B and make failover reliable, without rebuilding the queue manager. The error log on MI-B is under /MQHA/MI.QM/errors. MI-A is still powered off.
1. What do you check first, and what is the most likely reason MI-B cannot take over?
Read the worked answer
Start with dspmq -m MI.QM -x on both nodes to see the instance state, then read MI-B's error log under /MQHA/MI.QM/errors. Check the shared mount and ownership: ls -ld /MQHA /MQHA/MI.QM, id mqm on both nodes, and whether MI-B's mqm can create and lock a file in /MQHA. The UID mismatch (1001 versus 1002) is a strong candidate: the NFS export presents files owned by the active node's numeric UID, and MI-B's mqm cannot access them, so the standby cannot acquire the queue-manager lock. Also run amqmfsck against the shared mount to check locking and NFS behavior. Do not run crtmqm or delete queue-manager files; that would destroy the recovery path.
2. How do you restore service on MI-B and make automatic failover reliable?
Read the worked answer
Align the mqm UID and GID on both nodes to the same numeric values, preferably 1001 on MI-B, and correct ownership/permissions on /MQHA only as required. Confirm the NFS mount options are identical and support the required locking; rebuild or remount if the storage team's export is inconsistent. Then start MI-B with strmqm -x MI.QM. If the active node is still down, this should acquire the lock. For reliable automatic failover, both nodes must run the same MQ fix pack, mount /MQHA before MQ starts, and pass amqmfsck periodically. Test with a planned switchover using endmqm -s MI.QM from the active node, not by killing it, and measure takeover time against the five-minute requirement.
3. How do you verify that service is restored and that a second failover will work?
Read the worked answer
After MI-B is active, check dspmq -x, DIS QMSTATUS, and the status of all channels; put a persistent test message and confirm it survives a second switchover. Check that the old active node, when repaired, comes up as standby and does not try to run concurrently. Update monitoring to alert on standby state changes and on amqmfsck failures. Record the UID/GID pair in the build standard so the next node build cannot repeat the mismatch.