Tiering a 400 TB media archive and rescuing a render farm that is at the volume IOPS ceiling
A media company keeps 400 TB of source footage and rendered masters in one Object Storage standard bucket. Access analytics show 11% of objects are read in a given month, nearly all within 45 days of upload; older objects are touched only when legal holds are reviewed. Regulation requires each object to be kept for 7 years from creation and then deleted. Object versioning is currently off. The transcoding fleet is six BM nodes, each with one 2 TB Balanced block volume (10 VPUs); render jobs take 40 minutes and iostat shows sustained 100% I/O utilization on those volumes. Leadership wants storage cost down 40% without making renders slower.
1. Design the bucket lifecycle and versioning strategy. State what you would do, what you would deliberately not do, and why.
Read the worked answer
Tiering by age, because age predicts access almost perfectly here.
- Day 0-30: leave in Standard. Read rate is high and early reads should never incur retrieval fees.
- Day 30: rule moves objects to Infrequent Access. This matches the data - 89% of objects are cold - and IA's 30-day minimum storage duration begins exactly when objects stop being hot.
- Day 90: rule moves to Archive. Archive has a 90-day minimum duration, so transitioning earlier only creates early-deletion charges without saving time.
- Day 2556 (7 years): delete. Express retention as days in the rule; lifecycle rules are scoped by bucket and optionally by prefix or object-name pattern, so use prefixes that mirror the retention classes rather than one global rule.
Versioning. Enable it only if overwrite protection is a real requirement. Every version is a billable object, so pair it with a rule that deletes non-current versions after a short window. For the regulatory requirement, a retention rule (optionally locked) is the right control - it prevents deletion even by administrators, which a lifecycle policy does not. Do not confuse the two.
Do not rely on lifecycle to move data back to Standard. Archive objects must be restored (a job measured in hours) before download; if legal review needs same-day access, keep those prefixes in IA instead of Archive. Plan for IA retrieval fees on the 11% monthly read volume - the storage delta dominates, but confirm it with a cost report before committing.
2. Diagnose the render bottleneck and propose fixes ranked by cost and disruption. Give the numbers behind your reasoning.
Read the worked answer
Diagnosis. Balanced volumes deliver 60 IOPS/GB and 480 KB/s per GB, but with per-volume ceilings of about 25,000 IOPS and 480 MB/s. A 2 TB volume computes to roughly 122,000 IOPS, so the ceiling binds first - the volume is already running at the Balanced maximum, and adding VPUs beyond the Balanced maximum changes nothing. Buying a bigger Balanced volume will not help either.
Options, cheapest first.
- Check the workload before buying anything. Confirm the shape of the I/O. If the renders write large sequential frames, throughput is the constraint, not IOPS - and local NVMe on a DenseIO shape may beat any block volume for scratch space. Also check the attach type: paravirtualized attachments have higher per-I/O overhead; iSCSI-attached volumes are the standard choice for I/O-heavy nodes.
- Stripe across volumes. Four 512 GB Balanced volumes in a RAID-0 or LVM stripe aggregate four ceilings - roughly 100,000 IOPS - at close to the cost of one large volume. This is usually the best price/performance move.
- Change the performance tier. Higher Performance or Ultra High Performance raises the per-GB and per-volume ceilings, and Ultra High scales with VPUs (up to the 120 VPU range). This is simpler but costs more per GB than striping.
- Separate the tiers. Put scratch on the fast/striped volumes and keep final masters on Object Storage rather than block volumes - block storage for a 400 TB archive is the wrong tool and would dominate the bill.
Verify with iostat and the volume's own metrics after the change, and re-measure the 40-minute job - if it does not move, the bottleneck was CPU or the encode itself.