RAG, Fine-Tune, or Prompting Under a 1.5s Budget
A support-ops team must lift automated resolution from 62% to 78% on a 12,000-document help center that changes roughly 200 pages a week. They have 200 human-labeled resolved tickets, two on-prem L40S GPUs (48 GB each), 8 requests/s at peak, and a 1.5 s p95 latency budget for the whole request. A zero-shot 7B model scores 62%; the same model with a carefully written prompt and 4 few-shot examples scores 68%. A RAG prototype chunks documents into ~500-token pieces (about 60,000 chunks) and retrieves the top 5. Measured on their hardware: 1,800 prefill tokens/s per GPU at 2.5k context, and 30 ms per decoded token at batch 8. Fine-tuning the 7B on the 200 labels is also on the table; the last such run took 6 GPU-hours.
1. Classify the three levers by which failure they actually fix, and state which one the numbers rule out as a primary fix.
Read the worked answer
The three levers address different failure modes and are not substitutes:
- Prompting / few-shot changes behavior the model already has. It moved 62% to 68%, which is real but small - it teaches format and refusal discipline, not facts.
- RAG fixes knowledge: facts the model never saw, and facts that changed last week. With 200 pages changing weekly, no training-based approach can stay current; RAG is the only lever whose freshness cost is a re-index rather than a retrain.
- Fine-tuning fixes behavior: output schema, tone, tool-call formatting, domain-specific decision rules. On 200 examples it will not add reliable knowledge about 12,000 documents, and it will not know about last Tuesday's policy change.
What the numbers rule out is fine-tuning as the primary fix for a knowledge gap. 200 examples is also small enough that any measured lift will have a wide confidence interval - with roughly 200 samples, a 2-point improvement is indistinguishable from noise.
The defensible architecture is RAG for knowledge plus a light fine-tune (or better, structured prompting) for output discipline - subject to the capacity question, which is where the design actually breaks.
2. Can two L40S GPUs serve top-5 retrieval with 2,500-token contexts at 8 requests/s within a 1.5 s p95? Show the arithmetic.
Read the worked answer
Two independent bottlenecks, both binding.
Prefill. 8 req/s x 2,500 tokens = 20,000 tokens/s of prefill demand. Supply is 2 x 1,800 = 3,600 tokens/s. That is a 5.6x shortfall; the prefill alone would take ~1.4 s per request on one GPU, leaving nothing for decode.
Decode. At batch 8, one GPU produces 8 / 0.030 = ~267 tokens/s aggregate, i.e. ~1.33 requests/s for 200-token answers. Two GPUs give ~2.7 req/s against 8 req/s of demand - a 3x shortfall. And the 1.5 s budget only allows ~1,300 tokens of prefill plus decode on top, so the queueing behavior at 8 req/s offered load destroys p95 well before the average matters.
Conclusion: two GPUs cannot meet the SLO at peak with this design, regardless of whether retrieval is perfect. Levers, in order of value:
- Cut context: top-3 chunks of 150 tokens plus a 300-token system prompt is ~750 tokens, cutting prefill demand 3.3x to ~6,000 tokens/s - still above 3,600, so combine with (2).
- Cache: prefix-cache the system prompt and add a semantic cache for repeated intents; if 30% of traffic hits, effective prefill drops to ~4,200 tokens/s.
- Add capacity - roughly 4-6 GPUs for the full top-5 design at 8 req/s.
- Cap max output tokens; every 50 tokens saved is ~1.5 s of decode at these rates.
Whatever is chosen, state the SLO for the reduced design explicitly rather than assuming the original 1.5 s survives.
3. The team proposes fine-tuning on the 200 labels and reporting the improvement on the same 200. How do you make the measurement trustworthy, and what will you tell them about expected effect size?
Read the worked answer
Evaluate on examples the model never saw. Split the 200 into train/held-out (or, better, cross-validate, since 200 is small), and - critically - hold out by intent/topic, not at random, so near-duplicate tickets do not leak across the split. Random splits on support tickets leak badly because tickets cluster.
Additional guards:
- Freeze the eval harness before training: same prompts, same decode parameters, same retrieval (if any), scored blind.
- Report per-intent slices, not just the aggregate. A 3-point aggregate gain that comes entirely from one intent is a regression elsewhere.
- Have a second annotator score a 50-item subset to estimate label noise. If human agreement is, say, 85%, an automated metric cannot reliably resolve a 2-point difference.
- Compare against the strongest baseline, which here is the 68% few-shot prompt, not the 62% zero-shot one.
On effect size: with 200 examples, the 95% confidence interval on a measured accuracy is roughly +/-6 points. Tell them to expect the fine-tune to buy format compliance and consistency, not a leap to 78%. The 78% target requires the knowledge to be present at inference time - that is a retrieval problem, and it should be tracked as one.