Grounded RAG Service on Model Studio: Quality vs Cost
A support team is building a retrieval-augmented assistant on Alibaba Cloud Model Studio (Bailian). The knowledge base is 40,000 internal help articles. Requirements: answers must cite the source article; the assistant must not invent policy details; p95 latency target is 3 seconds; budget is tight. The team prototypes two configurations. Config A: embed all articles, retrieve top-3 chunks, and send them to a large, high-capability model with a strict 'answer only from context, otherwise say you don't know' instruction. Config B: retrieve top-10 chunks and send them to a smaller, cheaper model with the same instruction. In evaluation, Config A answers 71% of questions correctly with 4% unsupported answers. Config B answers 78% correctly but with 19% unsupported answers. A separate issue: some articles were updated last week but the assistant still quotes the old text.
1. Which configuration should ship, and what does the unsupported-answer rate tell you about the retrieval vs generation split?
Read the worked answer
Neither ships as-is; Config A is the right base, with the retrieval depth borrowed from B.
The critical metric is the unsupported-answer rate, not raw correctness. For a policy assistant, an invented answer is worse than 'I don't know.' Config B's 19% unsupported rate is disqualifying regardless of its higher correctness, because it indicates the model is filling gaps when retrieved context is weak. Config A's 4% is closer to acceptable but still needs work.
The split matters: B's higher correctness alongside a much higher unsupported rate suggests more retrieved chunks give the model more chances to find a real answer (good) but also more irrelevant context that invites confabulation (bad). The fix is not 'bigger model' or 'more chunks' alone; it is better retrieval precision plus grounding enforcement.
Recommended: keep the stronger model with the strict grounding instruction (Config A), increase retrieval depth moderately (e.g., top-5 to top-8) but rerank and filter chunks by relevance score before sending, and require the model to output a citation for every claim. Re-evaluate: target correctness above A's 71% while holding unsupported answers at or below A's 4%.
2. How do you address the stale-article problem and the 3-second p95 target together?
Read the worked answer
Stale answers are a data-pipeline problem, not a model problem. The embedding index must be updated when source articles change. Implement incremental re-indexing triggered on article update (or a short scheduled sync), and delete or supersede old chunks so the previous version cannot be retrieved. Verify by checking that a recently edited article's new text is retrievable and the old text is gone.
Latency: retrieval depth and reranking add time, so the 3-second p95 must be engineered deliberately. Levers: cap the number of chunks sent after reranking (rerank a wider candidate set but pass only the best few to the model); use a smaller/faster model for simple queries and escalate to the stronger model only when confidence or retrieval score is low; stream the response so perceived latency is lower; and cache frequent question-answer pairs. Measure p95 end-to-end (retrieval + rerank + generation), because optimizing only generation can miss the target. If the stronger model alone cannot meet 3 seconds at p95 with adequate grounding, a tiered routing approach preserves quality where it matters while keeping most requests fast.