Performance Tuning and Troubleshooting
Domain 3 - Performance Tuning and Troubleshooting
This domain tests structured bottleneck isolation on OceanStor arrays: reading counters correctly, separating front-end, cache and back-end limits, and choosing the fix that matches the measured bottleneck rather than the loudest symptom.
Method
- Establish a baseline with the workload's real I/O size and read/write mix; a 100% read synthetic test hides write-cache behaviour.
- Walk the path: application, host queue depth, HBA and MPIO, front-end port, cache, back-end disks. The first saturated component is the bottleneck.
- Read cache hit ratio, write-cache usage, front-end port bandwidth, disk utilization and per-LUN IOPS and latency in DeviceManager performance monitoring; SmartKit collects the same data for trend analysis.
- Watch tail latency (95th and 99th percentile), not averages. A good mean with a bad p99 is still a user-visible problem.
Tuning levers
- SmartQoS caps IOPS or bandwidth per LUN to protect latency for priority workloads; it does not add capacity.
- SmartPartition reserves cache for critical LUNs; SmartTier promotes hot chunks to SSD.
- Write-back caching improves write latency but depends on a healthy battery or capacitor; write-through is for integrity-first cases.
Worked example
An OLTP database runs 8 KB random 70/30 read/write and delivers 12,000 IOPS at 9 ms against a 40,000 IOPS target. Counters show front-end ports at 30% utilization, cache hit at 40%, and NL-SAS disks at 99% busy with a long average queue. The back end is the bottleneck: 8 KB random on 7.2k NL-SAS is seek-bound. Move the hot LUN to an SSD tier or add SSD capacity, then re-measure. Adding front-end ports or raising host queue depth would not help and would increase latency.
Traps
Blaming the array when the host HBA queue is full; comparing IOPS across different block sizes; applying QoS instead of fixing insufficient back-end IOPS; benchmarking during a snapshot or replication window and attributing the spike to the array.