Transcoding burst, 200 TB of media, and a finance objection
A short-video editing SaaS runs on Huawei Cloud. Editors upload 4K clips through an ELB in front of 6 ECS web servers. Transcoding runs on 40 ECS instances with 16 vCPU each; a worker executes 4 jobs concurrently, so the fleet is sized for a peak of 160 concurrent jobs of about 8 minutes each on weekday evenings between 19:00 and 23:00. Outside that window the workers sit at roughly 3% CPU and the job queue is empty. OBS holds 200 TB and grows by about 12 TB per month. Editors re-open clips from the last 30 days constantly - around 70% of all read requests - while anything older than 30 days is retrieved perhaps 20 times a year for compliance. Project metadata lives in a 400 GB RDS for MySQL instance that peaks at 8,000 IOPS with 25 ms write latency during the evening window. Finance objects to paying for 40 always-on workers.
1. Design the OBS storage plan for this data set, including the lifecycle rules, and explain the trade-off you are accepting at each tier boundary.
Read the worked answer
The data has three clearly separated access patterns, so a three-tier lifecycle is justified rather than a single standard bucket.
Tier 1 - Standard, 0 to 30 days. This holds roughly the last 30 days of growth plus whatever was uploaded before the cut-over, and serves ~70% of reads at low, predictable latency. Keeping this in Standard is not a cost failure; it is what protects the editing experience. The cost driver here is request count, not storage volume, so caching the hottest thumbnails and proxy files in DCS Redis or on a local EVS cache matters more than squeezing the storage class.
Tier 2 - Infrequent Access, day 31 to about day 180. Reads drop from constant to rare, which matches IA's shape exactly. The trade-off: IA carries a per-retrieval charge and a minimum storage duration (30 days for IA on OBS), so objects that get deleted at day 35 would be billed as if they lived 30 days anyway. At 12 TB/month growth, that penalty is small relative to the storage saving.
Tier 3 - Archive from about day 180, and Deep Archive for the compliance tail. The ~20 retrievals a year fit Archive comfortably, but retrieval is not instant and takes hours; a compliance officer waiting on a retrieval is fine, an editor waiting is not. Archive and Deep Archive also have longer minimum storage durations (90 and 180 days respectively - confirm against current OBS pricing before you commit, since these numbers change).
What I am deliberately not doing: moving the 30-180 day band to Archive to save more. It would break the occasional re-edit, and the retrieval cost at that read rate would partly cancel the storage gain. Verify the lifecycle policy against prefix tagging - a mis-scoped rule that archives the active prefix is the classic self-inflicted outage here.
2. Replace the 40 always-on workers with a scaling design. Be specific about policy types, timings and what happens to a job that is mid-flight.
Read the worked answer
The load is not random - it is a scheduled 19:00-23:00 weekday shape with an occasional backlog. That means the primary policy should be time-based, with metric-based scaling as the safety net, not the other way round.
Baseline. Keep 3-4 pay-per-use workers running around the clock for daytime interactive preview jobs and for queue drain after the window. This is cheap and removes the cold-start problem at 19:00.
Primary policy - scheduled. Use an AS scheduled policy to raise the desired count to 40 at 18:40 (a 20-minute lead, because a 16 vCPU instance takes minutes to boot and register) and to drop it at 23:30, not 23:00. Scaling in at exactly the window close strands queued work.
Safety net - metric policy on queue depth. Alarm-based scaling on DMS consumer lag or the pending-job metric, with a scale-out step that adds 25% capacity when lag exceeds one window's worth of throughput. CPU is the wrong metric here: transcoding is IO- and encoder-bound, and a worker at 3% CPU may simply be waiting on OBS reads.
Mid-flight jobs. A scale-in that terminates a worker running an 8-minute encode burns the job and the compute already spent on it. Two requirements: the worker must stop pulling new jobs on a termination signal and finish or checkpoint in-flight work, and the scale-in cooldown must exceed the maximum job duration (8 minutes) with margin - 10-12 minutes. Huawei Cloud AS supports instance protection, which should be applied to workers holding an active job.
Spot capacity. Interruptible transcoding is a strong fit for spot ECS, but only with checkpointing that lets a reclaimed worker's job resume on another instance rather than restart. Cap spot at a fraction of the fleet (60-70%) and keep on-demand capacity for latency-sensitive preview jobs, otherwise a spot reclaim wave shows up as an editor-visible stall.
3. RDS shows 8,000 IOPS and 25 ms write latency during the same evening window. Lay out your diagnostic order and your likely fixes, and say what you would not do first.
Read the worked answer
The instinct is to buy a bigger instance. That should be the last step, not the first, because 8,000 IOPS with 25 ms latency does not tell you whether you are at the storage ceiling or simply issuing bad I/O.
Step 1 - is 8,000 the ceiling? Check the instance class and the EVS storage type backing the RDS instance. If the provisioned IOPS limit for that combination is near 8,000, you are queueing at the device and the latency is arithmetic. If the ceiling is much higher, the problem is the query mix.
Step 2 - slow query and lock analysis. Look for missing indexes on the metadata tables that the evening job writes to, for long transactions holding row locks, and for write amplification from an update-heavy pattern. A single unindexed range scan executed 160 times per minute can consume the entire IOPS budget on its own.
Step 3 - connection behaviour. Confirm the connection count and whether the transcoding workers open a connection per job. 160 concurrent jobs that each open a fresh connection produce a connection storm that looks exactly like an IO problem at the storage layer.
Fixes, in order of cost: cache the hot project metadata in DCS Redis so the evening window stops hitting RDS for reads at all; batch the metadata writes from the workers into larger transactions; add a read replica if the 8,000 IOPS is read-dominated; only then move to a higher storage tier or instance class.
What I would not do first: vertically scale the instance. It costs money every hour of the month to solve a four-hour-a-day problem, and it will mask the query pattern rather than fix it - the latency will return as volume grows.