RAG assistant at 6.8 s p95 and $9,200/month: fix retrieval or fine-tune?
A support assistant answers product questions with a RAG pipeline: a 3,000-token system prompt, 12 retrieved chunks of about 500 tokens each, and a GPT-4o deployment that returns four sentences. Volume is 400,000 requests per month. Token spend is $9,200 and p95 latency is 6.8 seconds. Offline evaluation shows 78% answer relevance and 61% groundedness; about 30% of failures cite chunks from a different product line that shares vocabulary. Product wants p95 under 3 seconds and a 30% cost cut without reducing quality. The team has 20,000 resolved tickets with notes, and legal forbids sending customer PII to any service outside the existing DPA.
1. Where do the cost and latency actually come from, and what do you change first?
Read the worked answer
Do the token arithmetic before touching anything. Input per request is 3,000 (system) + 12 x 500 (chunks) = 9,000 tokens; output is roughly 80 tokens for four sentences - call it 100. So over 98% of spend is input tokens, and cost is dominated by the context you stuff in, not by what the model writes.
The same context drives latency: time-to-first-token grows with input length, and 9,000 tokens of prefill is a large share of a 6.8 s p95. Output length is already short.
First changes, in order:
- Cut retrieval count from 12 to 4-5 chunks - and use a reranker so the 4-5 are better than the current 12. Input drops to roughly 3,000 + 2,500 = 5,500 tokens before any prompt work.
- Compress the system prompt from 3,000 to ~800 tokens. Most such prompts are accumulated instructions and few-shot examples; move examples into the retrieved context or trim them. Combined, input falls to roughly 3,000 tokens - a ~65% reduction, comfortably past the 30% cost target, with latency improvement.
- Only then consider model changes.
Track cost per resolved request and groundedness together, because the cheapest failure mode is cutting context so aggressively that groundedness falls further.
2. 30% of grounding failures cite chunks from the wrong product line. Should the team fine-tune on the 20,000 tickets?
Read the worked answer
No - not for this symptom. Fine-tuning teaches behaviour and format, not facts, and it will not stop the retriever from returning a wrong-product-line chunk. If anything it makes things worse by letting the model answer fluently from parametric memory instead of citing the context.
The failures are a retrieval problem: same-vocabulary, different product line means pure vector similarity confuses near-duplicates. Fixes:
- Metadata filtering at query time: tag every chunk with product line and filter before/during retrieval, derived from the ticket's product field.
- Hybrid search (lexical + vector) with a cross-encoder reranker over a wider candidate set (retrieve 30, keep 4). Lexical signal separates model numbers and SKUs that embeddings blur.
- Chunk-level provenance and a hard rule in the prompt: if the retrieved chunks conflict on product line, refuse or ask a clarifying question.
Where fine-tuning would earn its place: making the model consistently refuse when grounding is absent, or matching a house answer format. That is a small, targeted dataset, not the 20,000 tickets dumped in.
Legal constraint: before either fine-tuning or indexing, strip or tokenise PII from the ticket notes - the index is content sent to the model provider, and the DPA covers the existing service, not a new corpus with raw customer identifiers.
3. How do you prove the changes helped, and how do you de-risk the rollout given the latency and cost targets?
Read the worked answer
Build the evaluation harness before changing anything, so the comparison is fair.
- Golden set from resolved tickets. Sample a few hundred tickets whose resolution notes are the ground truth, plus adversarial cases (cross-product-line questions, questions with no good answer). Score relevance and groundedness against the same judge configuration used today, so the 78%/61% baseline is reproducible. Keep a human-reviewed subset, because LLM-as-judge scores drift when the system prompt changes.
- Report as a vector, not a scalar: groundedness, relevance, refusal-when-ungrounded, p95 latency, and tokens per request. A change that improves cost by 40% and groundedness by nothing is not a win; one that trades 3 points of groundedness for 65% cost is a decision for Product, made with numbers.
- Ship behind a traffic split. Put the new pipeline in a second deployment on the same endpoint and route 10% of live traffic, watching p95, error rate, and escalation-to-human rate. Cost is only verifiable in production because retrieved-context length varies by query mix.
- Set the acceptance rule in advance: e.g. p95 < 3 s, cost down 30%, groundedness not below baseline minus 2 points on the golden set. Rollback is an immediate traffic shift.
Watch for a latency trap: dropping from 12 to 4 chunks can be offset if the reranker adds a network hop - measure end to end, not per component.