Restoring OLTP Latency During a Nightly Backup
A university data centre runs 200 VMware VMs on an OceanStor hybrid array. The array has 64 GB cache, a performance tier of 12 SAS SSDs, and a capacity tier of 48 NL-SAS drives; SmartTier is enabled. Every night at 01:00 a backup server reads 15 TB from production LUNs using eight parallel streams. During the backup, VM latency rises from 3 ms to 40 ms and the help desk receives complaints. The team has already enabled SmartCache on every LUN, including backup LUNs, to improve performance. No budget is available for new hardware this quarter. The storage administrator must restore the 5 ms latency target during the backup window without reducing the backup's completeness.
1. Diagnose the likely causes of the latency spike. Consider cache, tiering, and front-end/back-end contention.
Read the worked answer
The backup is reading 15 TB through eight streams while SmartCache is enabled on every LUN. Large sequential backup reads have little reuse, so they evict production data from cache and can cause cache thrashing; this alone can turn 3 ms into tens of milliseconds. SmartTier may also promote backup data to the 12 SAS SSDs, consuming the performance tier. On the back end, 48 NL-SAS drives cannot absorb eight concurrent streams plus production writes without queueing. The result is front-end latency from backend saturation and cache pollution, not a capacity problem.
The fix is to isolate the backup I/O and protect production cache, not to add disks.
2. Propose a configuration and scheduling solution using Huawei features, without new hardware.
Read the worked answer
Create a SmartQoS policy that caps the backup LUNs or backup host ports to a defined IOPS and bandwidth, starting at about 30% of measured backend capability. Use SmartPartition to reserve the majority of cache for production LUNs and exclude backup LUNs from SmartCache and from SmartTier promotion. Reduce backup streams from eight to two or four and schedule the full backup outside clinic hours. Better, use HyperSnap or HyperClone to create a point-in-time copy and back up from the copy, so production LUNs are not read continuously; this preserves completeness. If the backup server supports it, enable client-side dedupe and compression and throttling. Apply changes one at a time so the effect can be measured.
3. How would you verify the fix and what thresholds would trigger a rollback?
Read the worked answer
Verify with a baseline before and after: VM latency p95, array front-end IOPS, backend utilization, cache hit ratio, and backup duration. Run a pilot on one night with one host group, then expand. Acceptance: production p95 latency at or below 5 ms during the backup window, backup completes within its window, and the restore test still passes. Rollback triggers: latency remains above 10 ms for 15 minutes, the backup misses its completion deadline, or a restore test fails. Keep the previous QoS and cache policy documented so it can be reverted quickly.