Cost Spike on a Batch Pipeline
A nightly batch job runs on a managed instance group of 20 n2-standard-8 VMs, processing files for about 90 minutes starting at 02:00. Last month's bill shows the MIG running 24/7 because the autoscaler min size was left at 20. Monthly compute for the MIG is roughly $9,000. The workload is restartable, tolerates VM loss, and has no license or stateful dependency. The finance team wants a 60% reduction. The engineering lead proposes switching to Spot VMs to hit the target in one change.
1. Evaluate the Spot VM proposal against the 60% target and the workload's characteristics.
Read the worked answer
Spot VMs are discounted heavily (commonly ~60–91% off on-demand depending on machine type and region), so on paper they can hit 60%. But the proposal conflates two independent savings:
- Idle-time savings (the big one): the MIG runs ~22.5 hours/day doing nothing. Scheduling it to exist only during the 90-minute window eliminates most of the waste regardless of pricing model.
- Price-per-hour savings: Spot reduces the rate but introduces preemption risk.
Because the workload is restartable and tolerates loss, Spot is acceptable, but relying on it alone is fragile: a regional Spot shortage at 02:00 could stall the batch, and the 60% target is then missed. The robust answer combines scheduled scaling to zero outside the window with Spot for the running window, and keeps a small on-demand fallback.
2. Design the change and state how you would verify the savings without breaking the SLA.
Read the worked answer
Design:
- Set MIG min size 0, max 20; add a schedule (or Cloud Scheduler + MIG resize API) to scale to 20 at 01:55 and back to 0 at 03:45.
- Use a mixed instance policy or a separate Spot MIG so preemption falls back to on-demand capacity for the tail of the batch.
- Ensure the job is idempotent and checkpoints progress so a preempted VM resumes rather than restarts.
Verification:
- Compare billing export line items for the MIG across a full week before/after, not just the invoice total (other services can mask the change).
- Confirm batch completion time stays within the window for 5 consecutive runs.
- Track preemption events in logs; if the fallback fires often, revisit the Spot ratio.