Nightly Render Farm: Autoscaling, Anti-Affinity, and Shared Storage
A visual-effects studio runs a nightly render pipeline on 40 fixed Exoscale instances in CH-GVA-2. Each job is CPU-bound, averages six hours, and writes intermediate frames to a 2 TB block volume mounted on a single coordinator instance. At peak the queue holds 300 jobs; overnight utilization is 85%, but daytime idle cost is significant. The studio wants to cut compute spend by at least 40% without missing morning delivery. They also lost a coordinator host last quarter when its hypervisor failed, stalling the queue for three hours. Proposed changes include an Exoscale Instance Pool with CPU-based autoscaling, anti-affinity groups, and moving intermediate frames to Object Storage. The finance team will approve only if the design explains failure domains, storage consistency, and a hard monthly budget cap. The team has limited engineering time and prefers managed Exoscale features over custom orchestration.
1. Propose the compute and autoscaling design. Explain why CPU-based autoscaling alone is a poor fit for this queue.
Read the worked answer
Use an Exoscale Instance Pool of CPU-optimized instances in CH-GVA-2, not 40 fixed instances. Set a minimum of 4 workers to avoid cold-start delay and a maximum of 60 to enforce the budget cap. Enable anti-affinity so workers spread across physical hosts; this reduces correlated host failure but does not protect against a whole-zone outage.
CPU-based autoscaling is a lagging indicator here. Jobs run for six hours, so CPU stays high long after the queue is already deep, and scale-out arrives after the morning deadline is at risk. Drive the pool from queue depth instead: a small controller polls the job queue and calls the Exoscale API to set pool size. If native pool autoscaling only supports CPU or memory thresholds, use the API-based controller rather than pretending CPU is sufficient.
Use cloud-init to install the render agent and register the worker with the queue. On scale-in, stop assigning new jobs and let the worker finish or checkpoint its current frame. Keep the coordinator metadata in a durable store with backups, not on a single instance. This design directly addresses idle cost, burst capacity, and the prior coordinator failure.
2. How should intermediate frame storage be redesigned, and what consistency risks remain?
Read the worked answer
The 2 TB block volume is single-attach and makes the coordinator a single point of failure. Move intermediate frames to an Exoscale Object Storage bucket in the same zone. Workers write each frame as an immutable object keyed by job ID and frame number, then update a manifest or metadata record. The coordinator only tracks state; it does not serve frame bytes.
Use local instance scratch for temporary files, then upload completed frames with multipart upload and checksums. Enable lifecycle rules to abort incomplete multipart uploads after a short period, because abandoned uploads consume storage and are hard to spot.
Consistency risk remains because object storage is not a POSIX filesystem. Do not rely on rename or overwrite semantics. Use write-once keys, read-after-write checks where the application needs them, and a manifest that is finalized only after all parts are uploaded. If an object is missing, the worker should retry the upload rather than assume eventual consistency will fix it. For final assembly, read from the manifest and verify checksums. This removes the coordinator from the data path and makes worker failure recoverable.
3. What cost guardrails and failure-handling controls would you add before go-live?
Read the worked answer
Enforce cost with a hard pool maximum of 60 instances, instance tags per job or team, and budget alerts at 70%, 90%, and 100% of the monthly cap. Schedule the pool down to the minimum of 4 during daytime if overnight capacity is the only requirement. Add Object Storage lifecycle expiry for intermediate frames after seven days so scratch data does not become permanent cost.
For failure handling, make jobs idempotent and lease-based. A worker takes a visibility timeout, sends heartbeats, and releases the job on failure so another worker can retry. Limit retries to avoid poisoned jobs looping forever. Use anti-affinity to reduce host correlation, and run the coordinator as an active/standby pair with leader election or use a managed database for queue state. Monitor queue depth, worker age, and failed-job rate. Test a deliberate pool scale-in and a coordinator failover before the first production deadline. The key judgement is that autoscaling without drain semantics and idempotent jobs will save money but lose frames.