Data Ingestion and Preparation for Context
This domain tests how you turn raw sources into retrieval-ready records without losing fidelity, permissions, or lineage. The exam focuses on incremental ingestion, document parsing, chunking, metadata, deduplication, and quality gates before anything is embedded.
Ingestion rules
- Prefer Auto Loader for incremental file arrival; use schema evolution and rescued-data handling so malformed records do not break the pipeline.
- Use Lakeflow Declarative Pipelines or Delta Live Tables for dependency management and expectations.
- Keep bronze raw, silver normalized, and gold retrieval-ready. Do not embed directly from bronze.
- Make writes idempotent: deduplicate by source URI, content hash, and version, and apply MERGE instead of blind appends.
Preparation for retrieval
- Parse before chunking. Preserve headings, tables, page numbers, and section paths so citations remain meaningful.
- Chunk by structure first, then by size. Small overlap, roughly 10–15%, reduces duplicate evidence without doubling the index.
- Store metadata with every chunk: doc_id, chunk_id, source, version, effective_date, ACL tags, section_path, and content hash.
- Normalize whitespace and encoding, but keep original text for citation display.
Worked example
A corpus of 50,000 PDFs includes scanned pages and revised policies. Land files with Auto Loader into bronze, recording source path and ingestion time. Detect scanned pages and OCR them, then parse text with page boundaries. Chunk to about 700 tokens with 100-token overlap, preserving section headings. Deduplicate by content hash, keep the latest version per doc_id, and write silver rows with ACL tags inherited from the source system. Create a gold view that filters deleted and expired documents. Reasoning: OCR must happen before chunking; version and deletion fields prevent stale or revoked content from being retrieved.
Traps
- Chunking raw HTML or PDF bytes before parsing, which produces unusable fragments.
- Dropping page and heading metadata, making citations impossible.
- Embedding PII before redaction or access review.
- Ignoring document revisions and deletions, so old policy text remains retrievable.
- Letting schema drift break the pipeline instead of isolating rescued records.