Retrieval, Embeddings, and Knowledge Integration
The retrieval pipeline
Understand each stage: ingestion and parsing, chunking, embedding, indexing, query transformation, retrieval, reranking, and context assembly. Chunking strategy matters - semantic or structural splits with overlap preserve meaning better than fixed windows for prose, while tables and code need structure-aware handling. Metadata (source, date, permissions) enables filtering and citation.
Embeddings and search
Know that embeddings capture semantic similarity, so they help with paraphrase but can miss exact identifiers, codes, and rare terms. That is why hybrid search (dense + lexical/BM25) plus a reranker is the robust default. Understand that embedding model choice affects dimensionality and comparability; you cannot mix vectors from different models in one index without re-embedding. Approximate nearest-neighbor indexes trade recall for speed.
Query and context quality
Query rewriting, decomposition, and HyDE-style expansion improve recall on vague questions. Reranking a larger candidate set before truncating to the context budget improves precision. Always ground answers with citations and instruct the model to say when evidence is insufficient.
Traps
Questions bait you with "just increase top-k" (dilutes context, adds noise), "embeddings handle everything" (fails on exact IDs), or stale indexes after document updates. Watch for permission leakage - retrieval must respect the caller's access rights, not just relevance. Another trap: evaluating retrieval by eyeballing answers instead of measuring recall and precision.
Rule of thumb
Retrieve broadly, rerank sharply, and assemble narrowly: pull many candidates, rank them, and put only the best few in context with citations.