RAG Pipeline Returning Confident Wrong Answers Across Tenant Boundaries
A company built a RAG assistant over an internal knowledge base of 400,000 documents stored in Amazon S3, chunked at 1,000 tokens with 200-token overlap, embedded with a general-purpose text embedding model, and stored in an OpenSearch vector index. Documents belong to 30 business units, and access is controlled by department. Users report two problems. First, answers cite real documents that do not actually contain the claim (the citation exists but the passage does not support it). Second, a user in Finance received a snippet from a Legal-only document. Retrieval recall on a 300-question golden set is 0.71 at k=5, and answer faithfulness scored by an LLM judge is 0.63. The team wants to improve groundedness and fix the access leak without rebuilding the entire stack.
1. Explain the likely causes of the two failure modes and propose a prioritized remediation plan that improves recall and faithfulness while closing the access leak.
Read the worked answer
Failure mode 1: unsupported citations. The model is producing citations from retrieved context but nothing forces the cited passage to entail the claim. Causes: (a) chunks are too large and topically mixed, so a citation points to a document that is related but the specific sentence is elsewhere; (b) 200-token overlap duplicates content and inflates similarity scores; (c) the prompt asks for citations but not for verbatim supporting quotes; (d) a single embedding model is used for both short queries and long chunks, causing query-document asymmetry.
Failure mode 2: cross-tenant leak. The vector index is almost certainly filtered after retrieval or not filtered at all. If the search returns top-k globally and the application filters afterward, k results can be exhausted by out-of-scope documents, and any code path that skips the filter leaks. The fix is pre-filtering at query time using a department/tenant attribute stored as metadata on every chunk, enforced by the retrieval layer, not the prompt.
Prioritized plan:
- Close the leak first. Add tenant and department as mandatory metadata on every chunk. Enforce a server-side filter in the vector query, derived from the authenticated identity, never from user input. Add a deny-by-default test suite that asserts a Finance principal cannot retrieve Legal chunks. Log and alarm on any query without a tenant filter.
- Fix chunking. Reduce chunk size to 300-500 tokens with modest overlap; split on document structure (headings) rather than fixed windows. This improves precision and makes citations verifiable.
- Improve retrieval quality. Add a reranker over a larger candidate set (retrieve 30, rerank to 5). Evaluate a domain-tuned or instruction-tuned embedding model on the golden set. Measure recall@k and nDCG before and after each change.
- Enforce grounded answers. Require the model to quote the supporting sentence verbatim and to answer 'not found' when no passage supports the claim. Add a post-generation check that the quoted sentence appears in the cited chunk; reject and retry otherwise.
- Re-evaluate. Re-run the 300-question golden set with faithfulness and citation-support metrics. Track recall@5, faithfulness, and leak-test pass rate as release gates.
2. How would you design the evaluation and monitoring so that retrieval quality regressions and access violations are caught before users see them?
Read the worked answer
Separate the pipeline into measurable stages and gate each one.
Offline evaluation. Maintain a golden set of questions with labeled relevant chunks (for retrieval) and reference answers with required facts (for generation). Compute retrieval metrics (recall@k, MRR, nDCG) and generation metrics (faithfulness, answer relevance, citation support). Run on every change to chunking, embedding model, index settings, prompt, or model version. Use an LLM judge but calibrate it against a human-labeled subset; report judge-human agreement so the judge itself is trusted.
Access-control testing. Build an automated suite that impersonates each department and asserts that no retrieved chunk carries an unauthorized tenant label. Include adversarial cases: prompt injection asking to ignore filters, queries crafted to match Legal terminology from a Finance account, and direct API calls bypassing the UI. Treat any failure as a release blocker.
Online monitoring. Log per-request: query, retrieved chunk IDs with tenant labels (redacted as needed), rerank scores, generated answer, citations, and the post-generation citation-support check result. Track dashboards for retrieval score distributions, 'not found' rate, citation-rejection rate, and latency. Alert on distribution shifts (for example, mean top-1 rerank score dropping) and on any request where a retrieved chunk's tenant differs from the caller's.
Feedback loop. Capture user thumbs-down and 'citation wrong' reports, triage weekly, and promote confirmed failures into the golden set. This keeps the eval set representative as the corpus and query mix drift.
Change management. Version prompts, chunking configs, embedding models, and index snapshots together. A retrieval change should be reproducible and rollback-able. Canary new configurations on a small traffic slice with automatic rollback if faithfulness or leak tests degrade.