Troubleshooting and Performance Optimization
Domain 6: Troubleshooting and Performance Optimization
This domain tests a structured method for isolating storage performance and availability problems across host, network, and array, then optimizing without breaking resiliency. Expect metric interpretation and bottleneck-elimination questions.
Methodology
Define the symptom, scope, and baseline; check recent changes; isolate the layer; use array analytics and host counters; collect logs and a support bundle; change one variable at a time. “Slow storage” is a symptom, not a diagnosis.
Metrics and math
- Front-end metrics: IOPS, throughput, latency, queue depth, IO size, read/write ratio, and utilization.
- Little’s Law: queue depth ≈ IOPS × latency. Rising latency at stable IOPS usually means a saturated resource or a queueing bottleneck.
- Backend IOPS = front-end IOPS × (read% + write% × write penalty). RAID 5 ≈ 4, RAID 6 ≈ 6, RAID 10 ≈ 2.
- Check host HBA queue depth, multipath policy, filesystem alignment, VM sizing, and antivirus scans.
- Check fabric ISL oversubscription, CRC errors, BB credits, iSCSI retransmits, MTU mismatch, and zoning.
- Check array controller CPU, cache hit, port utilization, disk utilization, RAID rebuild, snapshot and replication load, and QoS settings.
Optimization levers
Wide striping, correct RAID for write mix, QoS to contain noisy neighbors, thin provisioning, compression/dedupe where supported, additional paths or ISLs, queue-depth tuning within vendor guidance, larger IO sizes for streaming, and workload rebalancing. Optimization must preserve availability and data protection.
Worked example
VMware latency spikes while array IOPS is moderate. The array FC port is at 95% utilization, the fabric ISL is 4:1 oversubscribed, and the ESXi host has eight paths but the PSP is Fixed, so one path carries most IO. Fixes: change to round robin after validating ALUA, add ISL capacity or rebalance, and recheck port utilization. Adding array cache would not fix a fabric/path bottleneck.
Traps that cost marks
- Blaming the array before checking host and fabric.
- Using average latency instead of peak or percentile latency.
- Tuning queue depth beyond vendor guidance.
- Changing several variables at once.
- Ignoring firmware bugs and known issues.
- Confusing throughput (MB/s) with IOPS.