Prepare data for retrieval augmented generation (RAG)
From documents to retrievable chunks
RAG quality is mostly a data-engineering problem. Ingestion parses files (PDF, HTML, Office), cleans text, and writes to a Delta table with metadata such as source, title, section, timestamp, and access tags. Chunking splits documents into passages; embeddings convert each chunk into a vector stored alongside the text. Databricks Vector Search builds an index from a Delta table (Delta Sync) or directly from vectors, and refreshes continuously or on a trigger.
What to get right
- Chunking: respect document structure such as headings, tables, and code. Typical chunks are 200-800 tokens with 10-20% overlap; smaller chunks improve precision, larger chunks improve context.
- Embedding consistency: embed queries with the same model and version used for chunks; changing models requires re-embedding and re-indexing.
- Metadata: keep source and access fields for filtering and citations.
- Hybrid search: combine vector similarity with keyword search for exact terms, IDs, and acronyms.
- Permissions: index permissions and underlying table permissions are separate; an app still needs data access governed in Unity Catalog.
Traps
Assuming an index is fresh after a table update without checking refresh mode; storing chunks without parent document IDs, which breaks citations; fixed character chunking that splits tables; embedding queries with a different model than the index.
Worked example
For a 500-page policy manual: parse to text, split by heading into sections, then into about 500-token chunks with 50-token overlap; store doc_id, section, and effective_date; embed with the same endpoint used at query time; create a Delta Sync index with continuous refresh and a metadata filter on effective_date. At query time, retrieve top-k, optionally rerank, and pass chunks with citations to the LLM.