PAI deployment: right-sizing an inference service for bursty traffic
A team has fine-tuned a Qwen model on PAI and needs to deploy it for an internal application. Traffic is bursty: near zero at night, peaks of about 40 concurrent requests during business hours, and occasional spikes to 120 during monthly reporting. Each request generates up to 1,500 output tokens. The team first deployed a single large GPU instance and saw queue times of 30+ seconds during peaks and idle GPU time at night. They are considering either scaling up to a much larger GPU or adding many replicas of the current instance. Their budget allows either a modest number of large instances or a larger number of small instances, not both at high counts. They also want to avoid cold-start delays when spikes arrive.
1. How should the team think about scaling and instance sizing for this workload?
Read the worked answer
The workload is latency-sensitive, bursty, and output-token-heavy, which shapes the answer:
- Output tokens dominate latency. At 1,500 output tokens per request, generation time is long regardless of instance size. Concurrency per instance is limited by memory (KV cache grows with concurrent sequences and sequence length). A single instance will queue badly at 40 concurrent requests.
- Replicas beat a bigger single GPU for throughput, because horizontal scaling adds independent KV-cache capacity and parallelism. A larger GPU helps per-request speed but does not solve the concurrency ceiling as cleanly.
- Autoscaling with a warm floor: keep a minimum number of replicas running to absorb baseline and avoid cold starts; scale out on queue depth or concurrent-request metrics rather than CPU. Scale-in should be slow to avoid thrashing.
- Handle spikes proactively: for the monthly reporting spike, pre-scale on a schedule rather than waiting for autoscaling to react, since cold start plus model load is slow.
- Reduce per-request cost: cap max output tokens, stream responses so users perceive lower latency, and batch or queue non-interactive reporting jobs separately from interactive traffic.
The judgement is that a single large instance is the wrong shape for bursty concurrency; a small warm pool plus autoscaling plus scheduled pre-scale fits the traffic pattern and budget.
2. The team wants to eliminate cold starts entirely by keeping 120 replicas running 24/7. Evaluate this.
Read the worked answer
This solves cold starts but is a poor trade:
- Cost: 120 replicas running all night at near-zero traffic wastes most of the budget. The peak occurs monthly; provisioning for it continuously is the most expensive possible design.
- Better approach: keep a modest warm floor sized to business-hours baseline, autoscale up during the day, and schedule pre-scaling ahead of the monthly reporting window. Accept a short, bounded cold-start only in truly unexpected spikes, and mitigate it with a queue that holds requests rather than dropping them.
- Queue instead of drop: a bounded queue with backpressure gives predictable behavior during spikes without paying for permanent overcapacity.
The right answer balances cost against a bounded latency risk, not eliminating cold starts at any price.