A10 GPU Capacity Contention Between Production and Development
A tenancy in the Germany Central (Frankfurt) region has a production compartment prod-app and a development compartment dev-ml. The production service runs an instance pool of VM.GPU.A10.2 instances that scales from 2 to 4 instances. The tenancy's regional service limit is 10 A10 GPUs, and each VM.GPU.A10.2 consumes 2 GPUs. At 09:10, the development team launches three VM.GPU.A10.2 instances for a training job, using 6 GPUs. Production is running two instances (4 GPUs), leaving no room for one more production instance that needs 2 GPUs. At 09:25, a traffic spike triggers the production autoscaling policy to add one instance, but the launch fails with a LimitExceeded error. The on-call engineer receives an alert from the load balancer showing elevated latency. The development team says the training job can checkpoint but needs 20 minutes. The production service has an RTO of 15 minutes for capacity events. The tenancy has no quota policies today, and all engineers have manage all-resources in the tenancy.
1. What should the on-call engineer do in the first 15 minutes? Explain the order of actions and why terminating all development instances is not the default.
Read the worked answer
- Confirm that the failure is capacity-related, not networking: check the instance pool autoscaling history and the Audit log for
LaunchInstancereturningLimitExceeded. 2. Use Limits, Quotas and Usage to see A10 GPU consumption by compartment and identifydev-mlas the holder. 3. Contact the development owner and ask them to checkpoint and terminate oneVM.GPU.A10.2instance, which frees 2 GPUs. Terminating all three is not the default because it destroys a checkpointable job and is not needed: one instance frees enough capacity for production to add one instance. 4. Retry the scale-out or let the pool retry, then verify the new instance reaches the load balancer backend set. 5. Open a service limit increase request for A10 GPUs in Frankfurt, but do not treat it as the 15-minute fix. 6. Communicate the RTO risk. Prefer terminate after checkpoint over stop, because a stopped instance may not reliably release GPU capacity.
2. Design a quota and compartment model that prevents recurrence without blocking production. Include the limitations of quotas.
Read the worked answer
Set a compute quota on dev-ml for the A10 GPU resource to 2 GPUs, which limits development to one VM.GPU.A10.2 instance. Production already has a maximum pool size of 4 instances (8 GPUs), so the regional limit of 10 leaves 2 GPUs for development. Remove manage all-resources at tenancy scope from engineers and grant manage instance-family only in the compartments they own. Quotas limit consumption but do not reserve capacity: they cannot guarantee that production capacity is available if the regional limit is exhausted by other compartments or tenancies. If production needs a hard reservation, use a separate tenancy for production or request a higher regional limit and keep the quota headroom. Also document that the quota resource name and scope must be verified in the console, because quota policies are enforced as maximums, not as priority for production.
3. What monitoring would you add so the team learns about capacity exhaustion before a production scale-out fails?
Read the worked answer
Add three layers. First, monitor service limit and quota utilization for A10 GPUs by compartment with a dashboard and a threshold alarm well below 100%. Second, create a Service Connector Hub rule that filters Audit logs for LimitExceeded and sends matching events to a Notifications topic and the on-call channel. Third, add an alarm on instance pool scale-out failures and on load balancer latency, since those are the customer-visible symptoms. Review the autoscaling history in weekly operations reviews and keep a runbook that names the compartment owners. A limit increase request should also be tracked as an operational task, not as a substitute for early warning.