The Answer Is in the Corpus but Never in the Top Five

Three symptoms, three different causes. Exact identifiers are a lexical problem: dense embeddings compress rare tokens into a semantic neighbourhood, so a chunk containing a specific part number can lose to a topically similar chunk. Numbers from the wrong column are a chunking problem: fixed 512-token windows cut tables across boundaries and detach header rows from values, leaving the model to read a plausible number with no way to know which column it belonged to. German queries returning English documents from the wrong market are a filtering problem: the embedding model is behaving correctly, the documents really are semantically similar, but nothing in the pipeline expresses same market, same language, current version. Raising top-k to 50 without reranking mostly adds noise and burns context, though it does confirm whether the gold chunk is reachable at all. Swapping to a larger embedding model is the most expensive hypothesis and addresses neither table structure nor metadata, so it should be the last experiment, not the first.

Do the cheap, reversible things first and measure each step. Order: structure-aware chunking that splits on headings and keeps table headers with every row group, targeting 300-500 tokens with overlap at semantic boundaries; metadata on every chunk (market, language, product family, version, effective date) applied as a SQL filter in Oracle Database 23ai before the vector search; hybrid retrieval combining vector distance with Oracle Text CONTAINS scoring so exact codes match lexically; then rerank the top 40-50 candidates down to 5. Measure with retrieval metrics, not only final answer accuracy: recall@50 shows whether the gold chunk is reachable, recall@5 and MRR show ranking quality, and a separate correct-answer-given-correct-chunk rate separates retrieval failures from generation failures. If recall@50 is high while recall@5 is low, ranking is the problem and hybrid scoring plus reranking fixes it. Justify re-embedding only if recall@50 stays low after chunking and hybrid search, which is the one case where the representation is truly the bottleneck.

Version answers at the data layer, not in the prompt. If chunks describing the old and new tariff both enter the candidate set, the chat model will merge them into a confident wrong answer. Store effective_from, effective_to and supersedes_id per chunk, filter on the query date in SQL so only currently valid versions reach the vector search, and keep superseded documents for audit while excluding them from retrieval. At generation time require citations to carry document ID and version, instruct the model to answer only from supplied chunks and to say not found otherwise, then test that behaviour with deliberately unanswerable questions, because models otherwise fill gaps. For tables, retrieve the whole table region with its header rather than isolated rows, and consider flattening tables to markdown at ingestion so row and column relationships survive tokenization. Add a regression gate: any ingestion, chunking or model change re-runs the 300-question evaluation before promotion to production.