Read Latency After a New Backup Host Joins the SAN

More ISLs would not fix this, and the evidence rules the ISL out.

• ISL utilization peaks at 41%. Congestion is not bandwidth shortage; a link at 41% has headroom, and adding ISLs moves traffic through the same shared buffers without changing the sink behaviour.
• The latency is localized to one 16-port ASIC group, and hosts on the other edge switch are clean. ISL congestion would degrade everything downstream of the ISL uniformly.
• txwait and credit-not-available rising on the port facing the backup host, with zero-buffer-credit events, is the signature of a slow-drain device: a receiver that accepts frames but does not return buffer credits fast enough. Its ingress buffers fill, the port stops granting credits, and the back-pressure spreads through the shared ASIC group.

The backup host is the obvious candidate: a single 8 Gb HBA ingesting large sequential reads, competing with production and possibly with insufficient queue depth on the host side. Its nightly job succeeding on schedule is not reassurance - it succeeds because it is allowed to take whatever it needs.

So: this is buffer-credit starvation, not oversubscription. The corrective action is on the endpoint and its placement, not on the core.

Measure to name the port, then split the fix by timescale.

Confirmation:

• show interface fc1/<x> and counters brief, on every port in the suspect ASIC group, looking at the rate of change of txwait and credit-not-available, not cumulative totals - cumulative numbers on a switch that has been up for a year are meaningless.
• Correlate the port with the highest rate against show flogi database and the zoning database; the backup host's pWWN should resolve to that port.
• Cross-check on the array side: host queue depth and outstanding I/O for the backup initiator. If the array shows a large outstanding queue to a host that is not draining it, the host is the sink.
• A port-monitor policy with thresholds on tx-datarate/credit loss will turn this from a manual hunt into an alert next time.

Next 24 hours (stop the bleeding without a change window on the core):

• Move the backup host's cable to a port outside the production ASIC group, or onto the second edge switch if a spare port group exists.
• Constrain the job: lower the HBA queue depth and throttle the backup stream so the initiator cannot outrun its own drain rate.
• Re-watch the rate counters on the sixteen affected hosts to confirm they fall the same night.

Next quarter:

• Give the backup host multiple HBAs across port groups so one host cannot concentrate pressure in one ASIC.
• Apply MDS QoS to classify bulk backup traffic into a lower-priority class with a bandwidth cap.
• Add slow-drain monitoring to the standard SAN health check so this is caught in hours, not by the array vendor two weeks later.

It reduces the blast radius on the array side but does not solve the problem, and it creates a new one.

What it does help with: fewer target ports means fewer array front-end ports receiving the slow-drain back-pressure, so the number of production paths sharing a buffer with the backup host may shrink. That is a genuine, if indirect, benefit.

What it does not help with: the bottleneck is the backup host's own credit return rate on the switch port it is attached to. Restricting targets does not make that port return credits faster; the ingress buffer still fills, and the congestion still spreads through the shared ASIC group to the sixteen co-located hosts.

What it makes worse: shrinking the target set increases contention on the remaining array ports and concentrically increases the per-port load - you may convert a credit problem into an array front-end queue-depth problem, and you lose the ability to spread backup load during the window.

The right test of any proposal here is: does it change the rate at which the slow device consumes and returns credits, or move it away from production sharing a buffer? Zoning changes neither. Port-group placement and host-side throttling change both.