Flash tier saturation after SmartTier relocation
A dual-controller hybrid array serves 500 hosts over 16 Gb FC. The system has 120 × 1.92 TB SAS SSDs in RAID 6 (4+2) and 60 × 8 TB NL-SAS in RAID 6 (8+2), giving roughly 150 TB usable flash and 400 TB usable capacity. The production working set is estimated at 45 TB hot, 120 TB warm, and 300 TB cold. SmartTier uses a 7-day access window and runs relocation every Sunday from 02:00 to 06:00. After last Sunday's relocation, Monday peak-hour p99 write latency rose from 1.8 ms to 9 ms, the flash tier reached 94% used, the capacity tier reached 61%, and cache hit ratio fell from 96% to 82%. Hosts report queue depth spikes, but link utilization is below 40% and controller CPU is 55%. No hardware errors are logged. The team wants to know whether to add flash capacity, tune tiering, or change cache settings before the next relocation.
1. Is the flash tier undersized for the 45 TB hot working set? Explain the 94% utilization and the cache-hit drop, and name two metrics you would collect before buying more flash.
Read the worked answer
The 45 TB hot set is only about 30% of the 150 TB flash tier, so raw capacity is not the primary reason for the latency. At 94% used, however, the tier has only about 9 TB free, which is below the 10–15% free-space headroom SSD garbage collection needs; write amplification and GC pauses can raise p99 latency even when the working set fits. The 94% also implies roughly 96 TB of non-hot data is sitting in flash. A 7-day access window on 500 hosts will classify much weekly warm data as active, and the Sunday relocation itself streams data into flash. The cache hit ratio fell because the relocation and promoted warm data displaced the skewed hot pages from the 512 GB cache; cache size is not the issue. Before adding flash, collect per-LUN read/write IOPS and reuse distance, SSD program/erase and write-amplification counters, tier occupancy by policy, and cache dirty-page and eviction counters.
2. Recommend a remediation plan for the next relocation. Include at least two configuration changes and one validation metric.
Read the worked answer
- Pin the latency-sensitive production LUNs to a dedicated flash pool or a high-priority SmartTier policy so relocation cannot demote them. 2) Exclude backup, scan, and archive LUNs from the 7-day access window; shorten the hot-window for general LUNs to 24–48 hours so weekly warm data is not promoted. 3) Cap relocation bandwidth or run it in a narrower off-peak window, and keep at least 10–15% free flash. 4) Partition cache so sequential relocation reads do not evict random hot pages. Validate with p99 write latency under 3 ms at Monday peak, flash utilization below 85%, cache hit ratio at or above 93%, and stable SSD GC/write-amplification counters. Add flash only if the measured hot set, not the 7-day promoted set, exceeds about 120 TB. If validation fails, compare a 48-hour window against a 7-day window to avoid demoting genuinely weekly hot data.
3. What evidence would confirm that tier relocation, rather than host load, caused the latency increase?
Read the worked answer
Correlate the latency onset with the relocation start and completion, not with a host IOPS increase. Compare per-LUN read/write IOPS and queue depth during the relocation window; if host IOPS is flat while p99 latency and SSD write amplification rise, relocation is the cause. Check cache dirty-page and eviction counters: relocation reads should show a surge in sequential cache fills and evictions of hot random pages. Look at SSD garbage-collection counters and free-block levels; at 94% used, a relocation burst can trigger GC that persists after the job ends. Finally, pause or throttle the next relocation and observe whether latency returns toward 1.8 ms. If latency stays high with relocation paused, investigate host queue depth, fabric congestion, or a back-end SSD fault.