Volumes iSCSI path loss after a ToR firmware upgrade
A three-node Nutanix cluster provides block storage to two external ESXi 7 hosts through Nutanix Volumes. Each ESXi host has two 25 GbE NICs connected to separate top-of-rack switches. The volume group VG-VMFS01 contains one 8 TB vdisk and is attached to both hosts' iSCSI IQNs. ESXi uses the software iSCSI adapter with round-robin and one target portal. After a top-of-rack firmware upgrade, esxcli shows only one active path per host, and VM latency rises to 60 ms. Prism shows Volumes latency at 1.5 ms with no disk errors, but each host reports CRC errors on one NIC. CHAP is enabled. The infrastructure team wants to disable CHAP and create a second volume group to restore performance. The business requires no downtime for the VMs during troubleshooting. What should you advise?
1. Is CHAP or the single volume group the likely cause? What should be investigated first?
Read the worked answer
CHAP is unlikely to be the cause. A CHAP mismatch normally prevents iSCSI login entirely or produces authentication failures, not one healthy path plus CRC errors on one NIC. Disabling CHAP would reduce security and would not clear CRC counters. The CRC errors point to the physical or link layer after the firmware upgrade: cable, SFP, switch port, VLAN, MTU, or NIC driver. A second volume group also does not create path redundancy for the existing datastore; it adds another target and more complexity. First confirm the supported Nutanix Volumes design: if the volume group is exposed through a single iSCSI data services IP, one target portal may be expected, and path redundancy is handled by the cluster rather than ESXi MPIO. If the design supports multiple portals, check vmkernel port binding and target discovery. In either case, fix the CRC source before changing CHAP or volume groups.
2. What network and iSCSI changes should be made to restore the missing path?
Read the worked answer
Work one host at a time. On the affected host, check esxcli network nic stats get -n vmnicX for CRC and alignment errors, then replace the cable or SFP and move to the other switch port. Compare switch interface counters and firmware on both ToRs. Verify the iSCSI VLAN, MTU, and that the vmkernel ports are on separate subnets if the design uses two. Confirm the software iSCSI adapter has the supported target portal and that port binding is configured as documented. Check esxcli iscsi session list and esxcli storage core path list to see whether the missing path is a login failure or a physical down state. Do not disable CHAP. After the link is clean, rescan the adapter and confirm two paths or the supported single-path design.
3. How do you validate the fix without VM downtime?
Read the worked answer
Put one ESXi host into maintenance mode and vMotion its VMs to the other host, so no VM is down. Fix that host's NIC or switch port, rescan, and verify path state, CRC counters, and datastore latency before returning it to service. Repeat on the second host. During the change, watch Prism Volumes latency, ESXi datastore latency, and VM disk latency; a 60 ms guest latency with 1.5 ms backend latency points at the path, not the cluster. Only after both hosts are clean should you consider any volume group change. If a second volume group is still required for capacity or isolation, add it after the path issue is closed and test with a non-production datastore.