GPU chargeback with reservations and idle capacity
A research platform runs 64 A100 GPUs across eight nodes for model training and inference. Monthly GPU compute is $118,000. Teams reserve whole GPUs through namespace quotas; average GPU utilization is 38%, and 22% of allocated GPUs are idle for more than six hours at a time. Small inference jobs request a full GPU because the platform does not expose slices. A shared queue exists but is optional. Finance wants to charge back GPU cost; research teams say chargeback will kill exploratory work. The platform team proposes time-slicing all GPUs to double utilization; the infrastructure lead worries about memory isolation and noisy-neighbor effects on latency-sensitive inference.
1. Is time-slicing all GPUs the right first move? Diagnose the idle cost and recommend a sequence.
Read the worked answer
Time-slicing all GPUs is not the first move. At 38% utilization, about 24.3 of 64 GPUs are busy and 39.7 are idle. At $118,000/64 = $1,843.75 per GPU-month, idle capacity is roughly $73,200/month. That is the opportunity, but time-slicing does not capture most of it. Time-slicing improves packing for bursty development jobs, but it does not provide memory isolation or guaranteed SM partitioning, so it can cause OOMs and p99 variance for latency-sensitive inference.
Sequence instead:
- Reclaim reservations idle for more than two hours back to a shared pool, with a notification and a short grace period.
- Segment workloads: inference gets MIG slices or a dedicated small-GPU pool; training uses the shared queue with gang scheduling and preemption; exploration uses a time-sliced pool.
- Expose slice sizes so small inference jobs stop requesting a full GPU.
- Measure memory and SM utilization separately before declaring 38% to be the whole story.
This attacks the 22% idle reservation problem directly and leaves time-slicing where it is safe.
2. Design a GPU chargeback model that changes behavior without suppressing exploration.
Read the worked answer
Use a hybrid of reservation charge and usage charge. About 70% of capacity is reserved and charged to teams whether used or not, because reservations are what create the idle cost. The remaining 30% is a shared pool charged by GPU-hour or slice-hour, with a per-team monthly cap and a visible showback report.
Give each team a small free baseline, such as 10% of its current reservation, for exploratory work. Charge above that baseline at cost, with no markup. Production inference is charged back; research experiments get showback first, then chargeback after two quarters. Publish rates and the allocation method before the first invoice. Add a reservation TTL: a reservation idle for more than two hours returns to the pool.
The key behavioral point is that unused reservations must cost something. If teams are charged only for usage, they have no reason to release a whole GPU they might need later. A reservation charge plus a shared pool gives them a cheaper way to keep optionality.
3. Quantify the idle-reservation opportunity and set a realistic utilization target with assumptions stated.
Read the worked answer
22% of 64 GPUs is 14.08 GPUs idle for more than six hours at a time. If those GPUs are idle 12 hours per day on average, idle GPU-hours are 14.08 × 12 × 30 = 5,069 per month. The hourly rate is $118,000 / (64 × 730) = $2.53, so the cost is about $12,800/month. Reclaiming half of that time saves about $6,400/month; reclaiming most of it saves closer to $12,000.
That is smaller than the $73,200 total idle estimate, because much idle time is short gaps rather than the long reservations measured here. Do not promise the larger number. Pair reclamation with MIG slices for inference: if 30% of full-GPU inference requests move to half-GPU slices, effective capacity rises without new hardware.
A realistic target is to raise average utilization from 38% to 55% within two quarters, worth roughly $20,000/month at current prices, with the savings verified against actual node count and reservations. State the idle-duration and slice-adoption assumptions in the business case.