Fifteen-Minute Latency Spikes on Horizon Pod C
A Horizon deployment runs five pods with 1,800 concurrent sessions. Pod C hosts 250 call-center users on non-persistent instant clones with Blast Extreme through Unified Access Gateway. Users report audio crackle and 3- to 5-second screen freezes every 15 minutes. Network RTT from the branch is 18 ms with 0.8% packet loss on one of two uplinks. Storage latency on the Pod C datastore spikes to 45 ms during the same intervals, and CPU ready is 8%. A backup job runs every 15 minutes against a file server that shares the same datastore LUN. The issue began after the golden image was updated with a new Horizon Agent and a new antivirus build. Other pods with the same agent but different antivirus build are unaffected. Thirty percent of Pod C users report the issue, all on the same branch-office VLAN.
1. Build a triage plan that separates client, network, storage, and host causes. What would you measure first, and what can you defer?
Read the worked answer
Work from the symptom outward and keep the layers separate so one noisy metric does not dominate the investigation.
- Client and protocol: collect Horizon Client logs and Blast Extreme statistics from two affected users and one unaffected user on the same VLAN. Look for retransmits, bandwidth drops, codec changes, and audio redirection events at the spike times. This is cheap and can be done first.
- Network: check the branch uplink that shows 0.8% loss, plus MTU, QoS markings, and whether the loss occurs only on the affected VLAN. Run a continuous ping and path test for 30 minutes to correlate loss with the 15-minute cadence. Compare a user on the other uplink.
- Storage and host: capture
esxtopduring a spike for DAVG, KAVG, GAVG, CPU ready, and network drops on the Pod C hosts. Correlate with the backup job window and the datastore LUN. CPU ready at 8% is worth watching but is not yet a smoking gun. - Guest: check antivirus scan schedules, exclusions for Horizon agent and demoted-application paths, and agent logs. The new antivirus build is a change that correlates with the start date, but it does not explain why only one VLAN is affected unless scans are time-aligned with the backup.
Defer vendor escalation until you have one synchronized timeline showing whether the spike starts in storage, the network, or the guest. Without that, a vendor case becomes a log exchange with no hypothesis.
2. Interpret the metrics. Which cause is most likely, and what test would confirm or eliminate it?
Read the worked answer
The 15-minute cadence is the strongest clue. It matches the backup job, and storage latency at 45 ms on the shared LUN is high enough to stall guest I/O and cause multi-second freezes. Blast Extreme audio and video also suffer when the guest cannot service timely updates, so the storage spike can explain the screen and audio symptoms without any protocol fault.
The 0.8% packet loss on one uplink is real but affects only one path. It could explain the 30% of users on the affected VLAN, but packet loss alone rarely produces a clean 15-minute periodicity. The new antivirus build could amplify I/O contention during scan windows, especially if Horizon agent exclusions are missing. CPU ready at 8% is elevated but below the level usually associated with 3-to-5-second freezes.
Most likely combination: backup I/O on the shared LUN causing storage contention, amplified by antivirus scanning on the same interval, with packet loss on one uplink making the affected VLAN feel it first.
Confirming tests: pause or redirect the backup for two intervals and observe whether the freeze disappears. In parallel, check esxtop DAVG during a spike and compare with a non-spike minute. Then test an affected user on the good uplink and an unaffected user on the bad VLAN. If the freeze follows the storage spike regardless of network path, storage is primary. If it follows the VLAN, network is primary.
3. What evidence would you collect before opening a vendor case, and what immediate mitigations would you apply without waiting for the vendor?
Read the worked answer
Evidence package. Build a single timeline with timestamps from all layers: client and Blast logs, UAG session logs, Connection Server events, esxtop samples for the Pod C hosts, datastore latency, backup job start and end times, antivirus scan logs, and network loss counters for both uplinks. Include the golden image build number, Horizon Agent version, antivirus version, and the exact first-report time. Add two affected and two unaffected user sessions from the same pod, plus one affected user from a different pod if one exists.
Immediate mitigations. Move or reschedule the backup so it no longer shares the datastore LUN with Pod C, or throttle it during business hours. Add antivirus exclusions for Horizon agent and instant-clone paths and stagger scan schedules away from the backup window. If the uplink loss is confirmed, fail traffic to the healthy uplink or open a carrier ticket. For the most affected users, test a different branch VLAN to confirm the network path.
What not to do. Do not roll back the Horizon Agent across the whole pod before the timeline is built; that removes a variable you may need. If a rollback is required, do it on one host or one pool with a small pilot and keep the logs. The vendor case should state the leading hypothesis and the tests already performed, not just 'users report freezes'.