Oracle Latency Spike After VMware Growth and Snapshot Changes
An HPE Alletra 9000 supports a two-node Oracle RAC cluster and 40 VMware VMs. The array has four 8 Gb FC ports connected to two fabrics; each host has four paths. Before a change, peak latency was 1.2 ms at 45,000 IOPS. Two weeks ago, the team added 20 VMware VMs, enabled 15-minute snapshots on six datastores, and moved a backup job to 10:00. Since then, Oracle users report 15-20 second stalls at 10:00, and array latency reaches 18 ms. InfoSight shows one FC port at 92% utilization, several hosts logging path failover events, and host queue depths pinned at 256. Array CPU is 48%, cache hit rate is 96%, and capacity is 68% used. The VMware administrator wants to add more paths; the Oracle DBA wants to increase queue depth; the backup team says the job must run at 10:00. The storage architect must find the dominant cause, propose a remediation plan, and define how to prove the fix without destabilizing RAC.
1. Prioritize the likely causes and list the evidence you would collect before changing queue depths or adding paths.
Read the worked answer
Prioritize fabric congestion and the backup/snapshot change first, not queue depth. The evidence points away from array CPU and cache: CPU is 48% and cache hit is 96%, while one FC port is at 92%. A single hot port can cause SCSI timeouts, host path failover events, and elevated latency even when the array is not saturated. The 10:00 backup and 15-minute snapshots add bursty write and metadata load; if the backup job uses the same few target ports or zones, it amplifies the hot port. Pinned queue depths at 256 are a symptom: hosts are queueing because I/O is not completing, not necessarily because the array needs a larger queue. Collect before changing: per-port and per-initiator IOPS, throughput, and latency; per-LUN and per-host queue depth; ESXi path policy and path state; Oracle AWR and wait events; snapshot schedule and change rate; backup stream count and target LUNs; SAN logs for RSCN, CRC, and timeouts; InfoSight history around the change. Do not add paths or raise queue depth until you know whether the bottleneck is target-port balance, zoning, or backup burst. Increasing queue depth can make latency worse.
2. Propose remediation in order of least risk and expected impact. Include specific settings but explain why.
Read the worked answer
Least-risk order. First, move the 10:00 backup to a non-peak window or throttle it, and stagger snapshots so six datastores do not snapshot at once. This is reversible and tests the burst hypothesis. Second, rebalance LUNs and host paths across the array target ports; if zoning sends most hosts through one port, spread them. Use the HPE-recommended path policy for Alletra 9000 and the OS, and keep one policy consistent; do not mix policies on the same host. Third, remove or shorten unnecessary snapshot retention and check whether snapshots are consuming cache or metadata. Fourth, adjust queue depth only after port balance is fixed. If the port is still 92% at peak, lowering per-LUN queue depth can reduce latency, but it may also reduce throughput; measure. Adding paths may help only if the new paths go to underused target ports. If they go to the same hot port, they increase congestion. Validation: compare 10:00 latency, per-port utilization, path failover count, and Oracle wait events for three business days. Roll back one change at a time.
3. If latency remains after scheduling and rebalancing, what escalation data and design changes do you recommend?
Read the worked answer
Escalate with evidence if latency remains. Open an HPE support case and attach InfoSight telemetry, array logs, SAN logs, host multipath statistics, Oracle AWR, and the exact change timeline. Ask HPE to verify firmware and known issues, port errors, and array back-end balance. Run a controlled I/O test outside production to separate Oracle, VMware, SAN, and array effects. Design changes to consider: separate Oracle and VMware zones so a backup burst cannot affect RAC; dedicate backup traffic to its own array ports; upgrade 8 Gb FC to 16/32 Gb if the fabric or array ports are the limit; add array ports and rebalance; or move the VMware estate to a scale-out Alletra Storage MP B10000 if growth is structural. Do not treat this as a single tuning parameter. If the business cannot accept a maintenance window, use a pilot zone and a canary host. The final proof is a sustained peak test at expected IOPS with latency under the service-level objective and no path failover events.