Context Management and Retrieval-Augmented Generation
What to know
RAG grounds answers in retrieved documents. The pipeline is: chunk documents, embed them, store vectors, embed the query, retrieve top-k candidates, optionally rerank, then assemble a prompt with the retrieved passages and the question. Candidates must understand that retrieval quality bounds answer quality - no prompt fixes missing context.
Key concepts: chunk size and overlap trade-offs (small chunks improve precision but lose surrounding meaning; large chunks preserve context but dilute relevance and consume tokens); top-k selection; reranking to improve ordering; metadata filtering; and citation/attribution so answers can be traced to sources. Also understand context-window management: summarizing or trimming old turns, keeping the most relevant passages, and placing the question after documents.
Traps
- Assuming a larger top-k is always better; irrelevant passages add noise and cost.
- Ignoring chunking strategy and blaming the model for poor answers.
- Forgetting to instruct the model to answer only from provided context and to say when the answer is absent.
- Losing citations by not asking for them or not preserving source IDs.
- Stuffing the whole corpus into the prompt instead of retrieving.
- Over-summarizing history until critical constraints vanish.
Rule of thumb
Design retrieval and prompt together: retrieve a focused set, label each passage with an ID, instruct "answer only from the passages; if not present, say so," and require the ID in the answer.