Fabrikam: RAG quality and latency in field support

Follow-ups: rewrite before you retrieve The vector query is built from a bare follow-up ('and for the 2021 model?'), which is semantically empty. Add a cheap rewrite step (a small model, not the answering model) that turns the last few turns plus the new question into a standalone query. Keep the original text as the keyword term too, so exact part numbers are not lost to paraphrase.

Ranking: hybrid, then rerank

• Vector + BM25 keyword retrieval combined with reciprocal rank fusion. Pure vectors miss exact identifiers (bulletin numbers, SKUs); keyword alone misses paraphrase.
• Apply the semantic ranker to the fused top ~50 and send only the best 5-8 chunks to the model. Sending 3 raw vector hits is the root cause of thin, uncited answers.

Metadata and filters Index model year, product line, doc type, language and validity window as filterable fields; when a question names a model, filter on it instead of hoping retrieval infers it. For 'current' questions filter on the validity window rather than trusting recency ranking.

Chunking 500-800 tokens with 10-15% overlap, each chunk prefixed with document title and heading path so a chunk is intelligible in isolation, and each carrying document ID, version and page.

Staleness A nightly full rebuild guarantees up to 24 h of wrong answers. Move to incremental indexing: push mergeOrUpload as bulletins publish, or use an indexer with a change-detection policy. Give superseded documents an end date and filter them out rather than deleting history you may need to cite.

Multi-language: use a multilingual embedding model into one index with a language filter, or per-language indexes if ranking quality matters more than operational simplicity.

Grounding is a pipeline property, not a prompt trick

1. Give the model retrieved chunks with stable IDs and require inline citations by ID plus an explicit 'insufficient evidence' path with an escalation option. The escape hatch matters: a model with no permitted way to say 'I don't know' will invent.
2. Post-validate deterministically: every cited ID must exist in the retrieved set. This is cheap, catches fabricated citations that no prompt eliminates, and converts '31% lack a citation' into a measurable gate that blocks the response.
3. Add a retrieval relevance gate: if the reranked top score is below a threshold, skip generation and return search results or a handoff. Many unsupported answers are answers to questions whose evidence was never retrieved.

Why not fine-tuning: fine-tuning shapes style and format. The missing facts live in the index, and tuning on 250k documents is neither affordable nor updatable nightly.

Fix the instruction set: 'Answer only from context; cite chunk IDs; if the context does not contain the answer, say so; do not use prior knowledge' plus two or three few-shot examples including a refusal example.

Proving it

• Build a labelled set of ~300 real questions with expected source documents.
• Score per change: groundedness, answer relevance, and retrieval recall@k (separately - if recall is low, no prompt fixes the answer).
• Run it in CI on every prompt, chunking or index change; a prompt edit that improves tone can silently cost 5 points of groundedness.

The 18% unsupported rate is a measurement problem first: without the labelled set you cannot tell whether a change helped.

Break the 4.8 s into stages first (rewrite, retrieval, rerank, model TTFT, model generation). Roughly half is model, so retrieval tuning alone cannot get there.

Real levers

• Use the small model for query rewriting; it does not need the answering model's quality.
• Retrieve 50, rerank, send 5-8 chunks. Fewer, better chunks cut prompt tokens, which cuts both latency and cost.
• Deduplicate near-identical chunks and truncate boilerplate headers - engineers' PDFs repeat the same legal footer in every chunk.
• Cap max_tokens; answers that ramble past the useful point are latency you paid for.
• Cache: a semantic or exact cache for the frequent question set typically absorbs a large share of 2,000 daily queries.
• Run the rewrite and any metadata lookup concurrently rather than in sequence.

Cosmetic but valuable Streaming does not reduce total time - it reduces perceived time to first token. Clarify whether the 3 s target is TTFT or full completion; if it is TTFT, streaming plus a faster first step may already satisfy the business, and you avoid an expensive redesign.

Check the network path: if the app, Search and the model are in different regions, cross-region round trips add fixed overhead per query. Co-locate them.

Set the target per stage so a regression is attributable, and re-measure after each change rather than shipping all levers at once.