Nightly Backup Causes Oracle Log Sync Spikes on All-Flash Block Storage
A four-node Oracle RAC cluster runs on an OceanStor Dorado 6800 V6 all-flash array over 16 Gb FC. Production uses 24 LUNs mapped through two fabrics; each database host has two HBAs and UltraPath. Normal read latency is 0.3–0.5 ms and log file sync averages 1 ms. From 01:00 to 03:00 a backup media server reads eight 8 TB LUNs with eight concurrent streams over the same two fabrics. During this window the array's front-end port utilization reaches 82%, redo-LUN write latency rises to 18 ms, and Oracle log file sync reaches 25 ms. No path failover, fabric CRC errors, or host HBA errors are logged. Latency returns to normal after the backup. SmartQoS is not enabled. Change management says a second FC HBA pair and new switch ports are available but cannot be installed for three weeks. The business wants the backup window kept at two hours.
1. Which layers would you examine first to separate a front-end SAN bottleneck from array back-end or host-side contention? Name the specific counters or logs you would compare, and state what pattern would point to each layer.
Read the worked answer
Start with a time-aligned comparison, not a single counter. On the array, collect per-front-end-port IOPS, bandwidth, average and maximum latency, queue depth, and port-busy time during 01:00–03:00. Also collect LUN-level read/write latency for redo and data LUNs, cache dirty ratio, back-end disk utilization, and any reconstruction or tiering activity. On the hosts, collect HBA queue depth, UltraPath path state, and Oracle wait events. On the fabric, check CRC, credit-zero, and discard counters.
Reasoning: if only the ports carrying backup reads are busy, per-port utilization is near saturation, and redo-LUN write latency rises while back-end utilization and cache dirty ratio stay moderate, the front-end path or queue is the constraint. If all LUNs on the array degrade, back-end utilization is high, or cache dirty ratio is elevated, look at back-end or cache. If host HBA queue depth is pegged while array ports are not, the host queue or multipath policy is limiting.
Aggregate 82% port utilization is not enough: a single 16 Gb port at 82% can queue while another port is idle. Because no fabric errors are logged, I would not start with a physical-layer fault, but I would still check credit-zero and SFP counters.
2. What temporary controls can be applied in the three-week window, and what permanent changes would you make?
Read the worked answer
Temporary controls: put a SmartQoS policy on the backup LUNs or backup initiators to cap IOPS or bandwidth; reduce media-server streams from eight to two to four and increase read size; move the backup start time away from the redo-heavy batch if the two-hour window still fits. If spare target ports exist, zone the backup media server through separate array ports. Use an array snapshot and let the backup server read the snapshot instead of the production LUNs. Verify UltraPath and HBA queue settings, and confirm that the backup streams are actually using the paths assumed.
Permanent changes: isolate backup traffic on dedicated FC ports or a separate backup network; use single-initiator, multiple-target zones; spread hosts and backup streams across front-end ports; apply QoS at both array and host; and use snapshot or clone offload so the media server does not read production LUNs. Size front-end ports for peak production plus backup, not production alone.
Do not simply add HBAs and expect relief: more host queues can increase pressure on the same array ports. Protect Oracle log sync first by capping backup traffic, then validate with a controlled test before the hardware change.
3. What evidence would make you reject front-end SAN congestion as the primary cause?
Read the worked answer
Reject it if per-port utilization is low on the ports actually used by backup but redo latency still spikes; if back-end disk utilization or cache dirty ratio is at its ceiling; if all LUNs, including those not read by backup, show the same latency; if host HBA queue depth is saturated while array ports are idle; if fabric credit-zero or CRC counters climb; if the array is doing reconstruction, tier relocation, or replication; or if the media server is reading through a different path than assumed.
Also reject it if Oracle log file sync rises without a rise in redo write latency at the array, which points to host CPU, log buffer sizing, or commit rate. The correct conclusion requires time-aligned correlation across array ports, LUNs, back-end, fabric, host, and database. If only the aggregate port chart is available, that is not enough evidence. A credible finding should identify the constrained resource, the queue that forms, and why latency returns when the backup stops.