Slow-Drain Host Port Behind a 32G SAN Port Channel
A Cisco MDS 9710 connects two MDS 9396S edge switches through a 2-member 32G SAN port channel. During nightly backups, hosts on the edge report 10-15 second SCSI timeouts. The storage array ports are 32G; most hosts are 16G. On the core, show interface fc1/1 shows credit not available and show interface fc1/1 counters shows txwait near 90% during the backup window. The ISL port-channel members are not saturated; average Tx rate is 40%. One host port, fc2/17, shows 0 Tx BB credits for long periods while its host queue depth is set to the maximum. No interface has logged link resets. Port-monitor is disabled. The fabric has 14 switches and 600 hosts. The team wants an evidence-based fix, not a blanket credit increase.
1. How do you determine whether the problem is ISL congestion or a slow-drain device? Which counters and commands support the conclusion?
Read the worked answer
Compare the ISL path with the host-facing port. ISL congestion would normally show high utilization or sustained credit shortage on the ISL members and would affect many hosts sharing that path. Here the ISL average Tx rate is only 40%, while fc2/17 holds 0 Tx BB credits for long periods and the affected host has maximum queue depth. That pattern points to a slow-drain device: the host or HBA is not returning buffer credits fast enough, so the edge port eventually stops accepting frames and backpressures the fabric.
Use show interface fc2/17, show interface fc2/17 counters, show interface fc1/1 counters, show interface port-channel <ISL>, and the trunk/VSAN mapping to compare. Look for txwait/credit-not-available on the host port, zero Tx BB credits, and whether the condition follows one host or many hosts behind the same ISL. Confirm the array port and ISL are healthy with interface counters and, where useful, fcping. Correlate the timeout window with backup traffic and the host HBA queue depth. Do not treat a high txwait counter on an ISL as proof of ISL congestion without checking utilization and the affected port set.
2. Design a port-monitor policy that detects slow drain without disabling production ports, and state the action and threshold logic.
Read the worked answer
Create a port-monitor policy scoped to host ports, not to the ISLs, so that normal ISL buffering does not generate false positives. Monitor credit-not-available/txwait and perhaps transmit discards or link-state changes. Set a threshold that is clearly abnormal for the environment, such as sustained credit-not-available or txwait above a defined percentage for several consecutive polling intervals, rather than a single spike. Use syslog and SNMP trap actions first; do not set an automatic err-disable action during the backup window unless the operational policy explicitly allows it.
Apply the policy to host-facing ports and verify with show port-monitor, show port-monitor policy, and the interface-level port-monitor status. Test the alert path by generating a controlled condition or by reviewing a known slow-drain port. The policy should identify and timestamp the offender so the host team can act, and it should preserve the interface for investigation rather than silently shutting it down.
3. What remediation would you apply to fc2/17 and the fabric, and how do you confirm the fix?
Read the worked answer
Treat fc2/17 as the likely source and work with the host team to reduce HBA queue depth or update the HBA driver/firmware to a supported setting; maximum queue depth on a 16G HBA behind a shared ISL can create prolonged credit starvation. If the host cannot be fixed immediately and the timeouts are affecting other hosts, isolate fc2/17 during the backup window as a containment step, then restore it after the host setting is corrected. Do not solve the symptom by adding ISL credits or enabling larger buffers everywhere; that can mask the slow-drain device and move the failure elsewhere.
After the host change, confirm that fc2/17 returns credits normally, that txwait and credit-not-available counters stop increasing during backup, and that the previously affected hosts no longer log SCSI timeouts. Check the ISL port-channel utilization remains normal and that name-server/logins for the host are stable. Keep the port-monitor policy in place to catch recurrence and document the queue-depth setting as part of the host build standard.