Scoping a Risk Assessment for a RAG Assistant With No Clean Ground Truth
An insurer is deploying a retrieval-augmented assistant that answers employee questions about internal policy documents. The corpus is 40,000 documents spanning 12 years; roughly 15% are superseded versions, and versioning metadata is inconsistent. There is no labeled evaluation set. The project lead proposes a risk assessment focused on 'hallucination rate,' measured by sampling 100 answers and having two reviewers mark them correct or incorrect. The compliance lead is more worried about the assistant citing a superseded policy and an employee acting on it. The CISO wants to know about prompt-injection risk from documents. Timeline is six weeks to launch; the risk assessment is budgeted at 10 person-days.
1. Critique the proposed 'hallucination rate' assessment. What does it miss, and how would you restructure the risk identification?
Read the worked answer
The proposed metric is not wrong, but it is measuring the wrong risk for this system.
Problems:
- 'Hallucination' is under-specified. For a RAG system the failure modes differ: fabrication (no source), misattribution (real source, wrong claim), stale retrieval (superseded doc), and omission. A single correct/incorrect label collapses these.
- No ground truth exists, so 'correct' depends on which document version is authoritative - the very thing in dispute. Reviewers may disagree systematically.
- It ignores the compliance lead's concern, which is the highest-consequence risk: an employee acting on a superseded policy. That is a retrieval and versioning risk, not a generation risk.
- It ignores the CISO's concern: untrusted document content can carry injected instructions.
- n=100 with 2 reviewers gives a coarse estimate with wide confidence intervals and no subgroup view (by document age, by policy area).
Restructure around failure modes mapped to harm, not a single score: (1) stale/superseded retrieval, (2) misattribution/citation error, (3) fabrication, (4) prompt injection via documents, (5) over-reliance by users. For each: likelihood, impact, detectability, and existing controls.
2. Given 10 person-days and six weeks, propose a prioritized assessment plan.
Read the worked answer
Prioritize by consequence × detectability, not by ease of measurement.
Tier 1 (must do, ~6 days):
- Superseded-document exposure test (~2 days): construct a targeted set of ~30 questions where a superseded and current policy both exist; measure how often the assistant retrieves/cites the stale one. This directly addresses the highest-consequence risk and doesn't need a full labeled set.
- Prompt-injection tabletop + sample test (~2 days): plant adversarial instructions in a test document; observe behavior. Cheap, high signal.
- Citation-faithfulness spot check (~2 days): for 50 answers, verify the cited passage actually supports the claim. Narrower and more reliable than 'is the answer correct'.
Tier 2 (if time, ~3 days):
- Metadata quality audit on the corpus: quantify the 15% superseded and inconsistent-versioning problem; this is a data risk with a clear remediation path.
Tier 3 (defer with documented residual risk, ~1 day):
- Broad hallucination sampling at n=100 - keep it as a baseline but don't let it crowd out Tier 1.
Explicitly out of scope for 10 days: comprehensive fairness or red-team coverage; record as accepted residual risk with a post-launch review date. Recommend a phased launch (e.g., read-only, no policy actions) to buy time.
3. How should the assessment handle the absence of a labeled evaluation set?
Read the worked answer
Absence of labels is a constraint to design around, not a blocker. Options, in order of preference:
- Construct a targeted evaluation set from known-answer cases: pick questions where the authoritative answer is unambiguous and version-controlled. This gives a small but high-quality set focused on the risk that matters (stale retrieval).
- Use document metadata as ground truth for retrieval: if the retriever returns a superseded doc when a current one exists, that's a defect detectable without human judgment on answer quality.
- Citation-faithfulness instead of answer-correctness: reviewers check whether the cited passage supports the claim - a narrower, more reliable judgment than 'is this the right policy answer'.
- Inter-rater calibration: have reviewers label a shared subset first, measure agreement, and document disagreements rather than averaging them away.
- Treat the unlabeled corpus as a known limitation in the risk register, with a plan to build a labeled set post-launch.
The key discipline: be explicit that the assessment estimates risk on a constructed sample, and state how that sample may differ from real usage.