GPUs idle 69% of the time and imagery nobody reads
Orbital Analytics runs GPU training on a GKE Standard cluster with a fixed pool of 96 a2-highgpu-1g nodes. GPU utilization averages 31% and saturates only during two four-hour windows each week; training jobs must finish within an eight-hour nightly window. A separate restartable batch-scoring job runs nightly on 40 n2-standard-16 nodes for five hours and must publish scores by 06:00. Storage holds 620 TiB of satellite imagery in a multi-region Standard bucket; 78% of objects have not been read in 400 days, but the ML team occasionally reprocesses a random 5% of the archive when a new model architecture lands. Finance wants the platform bill down 40% next quarter without missing the 06:00 deadline.
1. Right-size the GPU pool without putting the weekly saturation windows at risk.
Read the worked answer
The fixed pool pays for peak 168 hours a week and uses it for 8. Split by commitment class:
- Baseline, 24/7: roughly 25–30 nodes, matching the ~31% average plus headroom. Buy 1- or 3-year commitments on this slice. That is the only part of the spend that is genuinely predictable.
- Weekly saturating windows: cover with DWS Flex-start or calendar-mode reservations. These give a guaranteed start time for a bounded duration, which is exactly the shape of two 4-hour windows, and they do not require you to pay for the other 160 hours.
- Everything else: spot, with checkpointing to Cloud Storage every N steps so a reclaim costs at most N minutes. If reclaims land with a median of two hours of runtime, a 15-minute checkpoint interval wastes under 12% of GPU-hours against a 60–80% price reduction - the trade is clearly worth it, but measure the actual reclaim rate on this shape first.
Add node auto-provisioning with an explicit GPU quota limit so a runaway job cannot silently scale to the quota ceiling, and set resource requests so the scheduler bin-packs two jobs per GPU node where memory allows.
The wrong answer is committing to 96 nodes at a discount. You would be buying a discount on idle capacity.
2. The nightly scoring job must publish by 06:00. How do you make it cheaper without making the deadline a gamble?
Read the worked answer
First, establish what the job actually needs. 40 n2-standard-16 nodes for 5 hours is 3,200 vCPU-hours a night, about 96,000 a month. Measure whether it is CPU-bound, memory-bound, or I/O-bound before choosing a shape; if it is memory-bound, n2-highmem or an n2d equivalent is often cheaper per unit of work than more vCPU. The metric that matters is cost per 1,000 records scored, not price per node-hour.
Then play the deadline as a two-tier problem:
- Run the bulk on a spot node pool with a GKE Job and Kueue, sharded into ~200 independent tasks so a reclaim restarts one shard rather than the job. Sharding is what converts spot from 'risky' to 'fine'.
- Keep a small on-demand pool behind a higher PriorityClass as a deadline fallback. Drain the queue at 04:30; anything unfinished moves to on-demand and is guaranteed to land by 06:00.
Add a preemption-tolerant output pattern: tasks write to a staging prefix and a finalize step commits only complete partitions, so a partial run never publishes partial scores. Alert on queue drain progress at a rate that would predict a miss, not on job failure.
3. What do you do with 620 TiB of imagery where 78% is cold but occasionally reprocessed?
Read the worked answer
Do the arithmetic, then check the read pattern before acting.
620 TiB is 634,880 GiB. At roughly $0.026/GiB-month for multi-region Standard that is about $16,500/month. The cold 78% is about 495,600 GiB. Moving all of it to Archive at roughly $0.0012/GiB-month would cost about $595/month; Coldline at ~$0.004 would be about $1,980/month. Savings look enormous.
The trap is retrieval. Archive has a 365-day minimum storage duration and retrieval fees around $0.05/GiB, with hours of latency. The ML team reprocesses a random 5% of the archive - 5% of 495,600 GiB is about 24,800 GiB, and pulling that from Archive costs roughly $1,240 per event plus the latency, which is tolerable. Pulling 20% would cost about $5,000 a run, which is not.
So: move to Archive only after measuring actual reads. Enable Cloud Storage storage insights or access logging for 30 days, then apply lifecycle rules by age, not by guess. A safer first step is Autoclass on a copy of the prefix, which moves objects to colder classes automatically without you predicting the distribution, and keeps the 5% random reprocessing cheap.