Fabrikam HR assistant: 58% citation accuracy on a RAG pipeline

Ranking principle: fix where the error is introduced before where it is displayed. Prompt tuning cannot recover a fact that was never retrieved.

1. Retrieval quality (largest effect).

• Fixed 512-token chunks with zero overlap cut clauses mid-sentence and separate a policy's condition from its exception; tables and numbered lists are shredded. Use a chunking approach that respects structure (split by heading/paragraph, then sentences) with modest overlap, and store parent document id plus page for citation.
• Cosine-only over one embedding field loses exact-match tokens - policy numbers, dates, department names, benefit codes. Switch to hybrid (keyword + vector) with the semantic reranker: retrieve ~50 candidates, rerank, pass the top 4–5 to the model. Hybrid plus reranking is the single biggest lever for "right document, wrong chunk" style failures.

2. Stale-version collision (explains the amendment complaints). Both the old and the new version of an amended policy are in the index with near-identical embeddings; the old one often wins the top-3. Fix by deleting superseded documents via the indexer's data-deletion policy, tagging documents with an effective-date field and filtering at query time, and de-duplicating by document key. Do not solve this by re-embedding nightly - the nightly window is for corpus-level model changes, not for content updates.

3. Grounding (explains the 15% unsupported claims). Temperature 0.7 plus "answer helpfully" invites invention. Set temperature to 0–0.2, instruct the model to answer only from the supplied passages, to quote the supporting sentence, and to say the information is not available when the passages do not contain it. Return structured output containing answer, cited chunk ids, and the quote, so citations can be validated automatically.

What I would not do first: swapping to a larger chat model. It raises cost and latency and will not fix either retrieval or versioning.

Separate the two failure modes and measure them independently. An end-to-end "is the answer good" score conflates retrieval failure with generation failure and tells you nothing about which fix worked.

• Retrieval evaluation: build a golden set of 250–300 real questions, each labeled with the document (and ideally the passage) that contains the answer. Measure recall@k and mean reciprocal rank. This runs in seconds and does not need a model judge.
• Groundedness evaluation: for the same questions, score whether every claim in the answer is supported by the retrieved context, and whether the cited document matches the label. Use an LLM judge against a fixed rubric, then have a human review a random 10% of judge verdicts to keep the judge honest. Track the judge's agreement rate over time.
• Refusal correctness: include questions whose answers are genuinely not in the corpus, and measure whether the system says so instead of inventing an answer. A pipeline tuned only on answerable questions learns to always answer.
• Regression gate: run the golden set on every prompt, chunking, or index change in a pre-production environment and block promotion on a drop in recall@k or groundedness. Version the golden set alongside the code.

Judgement point. 150 hand-labeled questions you trust beats 5,000 auto-generated ones you don't; auto-generated question/answer pairs usually leak the source chunk's wording and inflate retrieval scores. Seed the golden set from real support tickets and from the questions users asked in the last six weeks.

Sample size. With ~300 questions, a rise from 58% to 75% correct citations is comfortably outside noise; judging a 3-point improvement on 30 questions is not.

Diagnose first - do not guess. Instrument each stage separately: query embedding, search (split keyword vs vector vs semantic reranker time), prompt assembly, model first-token, model full completion, and any retry loops. Eleven seconds is not one number; it is typically 0.1–0.3 s embedding, 0.3–1.5 s search including reranking, and the rest in generation.

Check for throttling before optimizing anything. If the deployment is returning 429s and the SDK is retrying with backoff, latency is a quota problem, not a design problem. Look at the model's latency and throttling metrics over the same window as the complaint. This is the most common cause of a sudden multi-second median.

Then, in order of effort-to-benefit:

1. Stream the response. Perceived latency drops to time-to-first-token; for a chat UI this alone solves most of the complaint.
2. Reduce input and output size. After reranking, prompt with 4–5 tight chunks instead of 3 bloated ones; cap max_tokens; add "be concise" with an explicit length bound. Generation time scales with output tokens.
3. Cache. Cache embeddings of repeated queries, and cache full answers for the high-frequency questions (HR assistants have a long tail plus maybe 100 very common questions) with a short TTL and invalidation on document change.
4. Right-size the model per step. If you added a model-based query rewrite or a judge step, run it on a smaller deployment; reserve the large model for the final answer.
5. Trim the semantic reranker from stages where it adds little (e.g., very short queries) - measure before removing, since it is usually worth its cost.

Do not degrade retrieval quality to buy latency before the first four items are done; the current 58% citation accuracy is the bigger business problem.