VDI Boot Storm Causing CVM and Host Contention
A 5-node AHV cluster with 2 x 24-core CPUs, 512 GB RAM, 8 x 1.92 TB SSDs, and 25 GbE per node runs 850 non-persistent Citrix desktops. Between 07:00 and 08:00, Prism Central shows storage latency rising from 3 ms to 40-80 ms, CVM CPU at 85-95% on all nodes, host CPU at 70%, and about 90,000 IOPS dominated by 4K random reads. One host reports intermittent receive drops on its 25 GbE bond. A new endpoint antivirus full scan starts at 07:00 on all desktops, and Citrix logons take 3-5 minutes. Users report that after 08:30 performance returns to normal. The team has not changed the storage container, snapshot schedule, or AHV network configuration in six months. They need to determine whether to add nodes, reschedule tasks, fix the network, or open a support case.
1. Interpret the metrics: what is the likely bottleneck, and what additional data would you collect to confirm before adding nodes?
Read the worked answer
90,000 IOPS of 4K random reads is only about 360 MB/s, so this is an IOPS/queueing problem, not bandwidth. CVM CPU at 85–95% on all nodes is the strongest signal: the controller VMs are CPU-saturated processing I/O. Host CPU at 70% is not yet the bottleneck, and one host with receive drops can add retransmits but does not explain all five CVMs being high.
Collect before adding nodes: per-host CVM CPU and CVM CPU ready; storage controller latency and queue depth; SSD latency/wear; per-host network drops/errors; AHV VM CPU ready; Citrix logon phase breakdown; antivirus scan IOPS; and whether snapshots or dedup/compression are active. Use Prism Central performance charts and NCC. If CVM CPU is high while SSD latency is low, it is controller CPU; if SSD latency is high with low CVM CPU, it is the storage devices. If only one host is bad, investigate that host's network or SSD first.
2. Propose a remediation plan in phases, with expected impact and validation. Include Citrix, AV, AHV/storage, and network actions.
Read the worked answer
Phase 1, no hardware: stagger Citrix boot/logon with Citrix policies and randomize antivirus full scans away from 07:00; exclude the base image or use a low-impact scan profile. Expect the 90k IOPS peak to drop. Phase 2: fix the one host's 25 GbE bond-check LACP/switch logs, SFP/cable, and flow control; validate zero drops under load. Phase 3: if CVM CPU remains >85% during normal boot, add 2 nodes to distribute CVM CPU and SSD queues, or add SSDs if per-node disk latency is the limit. Keep RF2 for VDI; do not add erasure coding to active desktop containers. Separate VDI and server workloads into different containers only if it helps snapshot/retention isolation, not as a performance fix by itself.
Validate with a 7-day Prism trend: peak IOPS, 95th percentile latency, CVM CPU, host network drops, and Citrix logon time. Change one variable at a time so you can attribute the improvement.
3. When would you open a support case versus continue internal optimization? What artifacts would you collect?
Read the worked answer
Open a support case if, after rescheduling AV and fixing the network bond, CVM CPU remains >90% on all nodes at moderate load, if NCC reports failures, if latency is isolated to one host with no workload explanation, or if you suspect an AOS bug (for example latency without IOPS). Also escalate if a node cannot sustain its SSD or network reliably.
Collect: NCC report, Prism Central and Prism Element logs, AOS logs from affected CVMs, ncli/acli output for storage and network, per-host performance charts for the window, Citrix and AV logs, and change history. Internally you can fix workload shaping, bond issues, and capacity. Do not open a case before you have a clean before/after comparison; it will be routed back for data.