NAS antivirus scan metadata storm
A scale-out NAS cluster with four nodes and 480 TB usable capacity serves 2,000 SMB clients. Each node has 2 × 25 GbE front-end ports and 2 × 100 GbE back-end ports. The workload is 30% large CAD files and 70% office documents. Before a new endpoint-protection rollout, peak throughput was 4 GB/s and SMB p95 latency was 35 ms. The security team now runs a full on-access plus scheduled antivirus scan from 01:00 to 05:00 across all shares; scans open every file and read the first 256 KB. Since then, peak p95 latency is 220 ms, aggregate throughput has fallen to 1.2 GB/s, NAS node CPU is 95%, and metadata operations are 6× baseline. Snapshots run every 4 hours with 14-day retention. Network utilization is below 30%, and disk utilization is 45%. The business wants to keep daily scanning.
1. Diagnose the bottleneck and explain why adding a fifth NAS node is unlikely to restore 4 GB/s by itself.
Read the worked answer
The bottleneck is NAS metadata and file-open processing, not network or disk. Aggregate throughput fell from 4 GB/s to 1.2 GB/s while network is below 30% and disks are at 45%; node CPU is 95% and metadata operations are 6× baseline. The antivirus scan opens every file and reads the first 256 KB, creating a metadata-heavy, small-read workload across all shares. Adding a fifth node can add CPU and metadata capacity, but it is unlikely to restore 4 GB/s by itself because the scan will still hit every file, and the 2,000 clients may not rebalance evenly. If the cluster is not rebalanced, the new node may be idle while existing nodes remain saturated. First reduce the metadata work per file, then add capacity only if measured per-node CPU remains above 80% after scan optimization.
2. Recommend mitigation that preserves daily scanning and snapshots. Include at least one change to scan architecture and one to snapshot scheduling.
Read the worked answer
Move scanning off the live data path: scan snapshot clones or a dedicated scan share, use changed-file scanning after the first full scan, exclude CAD scratch/temp and known binary formats, and rate-limit scan threads per node. If on-access scanning is required, restrict it to writes or high-risk shares rather than all files. Schedule snapshots outside 01:00–05:00, or create the scan snapshot at 00:30 and let the scan run against that immutable snapshot; this also reduces snapshot metadata churn during the scan. Distribute scan windows by department instead of a cluster-wide 4-hour burst. Validate with node CPU below 70%, metadata ops near baseline, SMB p95 latency below 50 ms, and throughput above 3.5 GB/s during the scan. If those targets fail, add a node only after confirming the cluster can rebalance metadata.
3. How would you distinguish a metadata CPU bottleneck from a disk or network bottleneck using the given counters?
Read the worked answer
Disk at 45% and network below 30% rule out those as primary saturation points, while CPU at 95% and 6× metadata ops point to metadata. Confirm with per-node SMB operation latency: if create/stat/open latency rises much more than read/write latency, metadata is the constraint. Check disk read IOPS and average request size; the scan reads only 256 KB per file, so a disk-bound scan would show high IOPS and queueing, which is absent. Network would show high interface utilization and TCP retransmits, also absent. Finally, pause the scan briefly; if CPU and metadata ops fall and latency improves while disk and network stay similar, the scan-driven metadata path is the cause. If CPU stays high with the scan paused, inspect snapshot deletion or client rebalance activity.