Recall@5 of 41%: Contracts the Retriever Cannot Read

Diagnose before touching chunk size. Take the failing questions and hand-inspect the retrieved chunks against the source PDF. Three distinct failure modes are almost certainly present, and they need different fixes:

• Extraction failure. Scanned pages producing empty or garbled text; tables collapsed into a single line of numbers. If the text was never extracted correctly, no chunking or embedding change can retrieve it.
• Chunking failure. A clause split at a page boundary yields two chunks, neither of which contains the answer. Fixed-character splitting cannot know where a clause ends.
• Retrieval/match failure. The chunk contains the right text but the query ("what's the penalty for late delivery on the Acme contract?") does not lexically resemble dense pricing prose.

Fix order: extraction → structure → retrieval.

1. Parsing. OCR the scanned fifth with a layout-aware model; serialize tables to markdown so header rows stay attached to their data; detect page breaks and rejoin continuation text.
2. Structure-aware chunking. Chunk on section and clause boundaries with a target size, and prepend the heading path to each chunk ("Acme MSA > §7 Pricing > Table 7.2") so the chunk carries its own context. Keep small tables atomic rather than splitting mid-table.
3. Retrieval. Add keyword/hybrid matching alongside dense vectors - exact part numbers, currency figures, and contract names are precisely where embeddings underperform. Retrieve k=20 and rerank to 5.

Avoid guessing by measuring each stage separately: extraction success rate, then retrieval recall with the retrieval isolated from generation, then end-answer quality. Change one stage at a time against the same question set.

Stop treating the index as the source of truth. The Delta table should be the authoritative, versioned store; the index is a derived projection that syncs from it.

Pipeline shape:

1. Landing → parsed → chunked, all as Delta tables with doc_id as a stable key and a content_version/ingested_at column. Reparsing a document replaces its rows via a MERGE on doc_id (plus chunk ordinal), so a changed contract doesn't leave orphaned chunks from the previous version behind.
2. Enable Change Data Feed on the chunk table - a Delta Sync index requires CDF to pick up inserts, updates, and deletes incrementally. This is what converts a six-hour full rebuild into a minutes-long sync.
3. Trigger on arrival, not on a nightly window. A scheduled job (or file-arrival trigger) handles the ~30 documents/week, merges them, and the index syncs the delta. Deletes propagate from the source table, which is the operator you most need to be reliable - a withdrawn contract must disappear from answers.
4. Keep the full-rebuild path as an escape hatch for schema changes (new embedding model, changed chunking strategy). Those legitimately require a re-embed of the corpus, so budget for it and run it against a new index with an alias swap rather than mutating the live one.

Watch the embedding-model coupling: changing chunking or the embedding model invalidates every vector. Version both, and treat them as a schema change with a staged cutover.

Metadata to attach at parse time (each serving a distinct job):

• doc_id, content_version, chunk_ordinal - identity, dedupe, and correct replacement on re-ingest.
• section_path / clause_id - human-verifiable citations and heading-aware context.
• page_start, page_end - citation and source-highlighting.
• effective_date, expiry_date, supersedes - so a superseded contract can be filtered at query time instead of competing with its replacement.
• supplier, contract_type, region - metadata pre-filters that cut the search space before vector comparison.
• extraction_confidence / is_scanned - flags chunks that need review rather than silent trust.

Measure, sliced, not just overall:

• Recall@5 on the original 60 questions re-sliced into table questions, cross-page clause questions, and scanned-document questions - the original aggregate hid the failure clusters, and it will hide the fixes too.
• Freshness lag: publish time → time the answer is retrievable. That is the actual commitment legal cares about; measure it from the pipeline's timestamps, not from a job's success status.
• Extraction success rate per document type, with parse failures surfaced rather than silently producing empty text.
• End-answer correctness with citations checked against the cited chunk - a rise in recall that produces confident wrong citations is not progress.