Model Studio application with retrieval, guardrails, and cost control
A financial-services firm is building an internal assistant on Model Studio. It must answer questions over 200,000 internal documents, some of which are restricted to certain departments. The compliance team requires that answers cite sources and that restricted documents never appear in responses to unauthorized users. During a trial, the assistant answered a question about a restricted HR policy by quoting it verbatim to a user outside HR. The team also noticed that average cost per query is high because every query retrieves 20 chunks and sends a long conversation history. They want to keep quality high for complex questions while reducing cost for simple ones. They have not yet implemented any access control in the retrieval layer.
1. Diagnose the root cause of the restricted-document leak and describe where access control must be enforced.
Read the worked answer
The leak is a retrieval-layer authorization failure, not a prompt problem. If a restricted document is retrieved and placed in the model's context, no system-prompt instruction can reliably prevent it from being quoted. Prompt-level 'do not reveal' rules are a soft control and will fail under adversarial or even ordinary phrasing.
Access control must be enforced before retrieval results reach the model:
- Attach document-level and chunk-level metadata (department, classification) at ingestion.
- At query time, resolve the user's identity and entitlements, and apply a metadata filter to the vector/keyword search so unauthorized chunks are never candidates.
- Enforce the same filter in any re-ranking or fallback keyword search path; a common bug is filtering one retriever but not the other.
- Log retrieved document IDs per query for audit, and add a post-generation check that cited sources are within the user's entitlement set as defense in depth.
The compliance requirement ('cite sources') is actually an asset here: citations make it possible to verify that every claim traces to an authorized document. The team should treat citation enforcement and access control as one pipeline.
2. Propose a cost-reduction strategy that preserves quality on complex questions.
Read the worked answer
Use query routing and context discipline rather than uniformly shrinking the pipeline:
- Classify query complexity with a cheap step (rule-based or small model). Simple FAQ-style queries go to a small model with top-3 chunks and no history; complex multi-hop queries go to a larger model with more chunks and history.
- Reduce retrieved chunks adaptively: retrieve 20, re-rank, then pass only the top-k above a relevance threshold. Sending 20 chunks for every query wastes tokens and can dilute the answer.
- Trim conversation history: summarize older turns and keep only the last few verbatim. Long histories are a major hidden cost.
- Cache: cache answers for frequent policy questions, keyed by query and user entitlement set (never cache across entitlement boundaries).
- Measure: track cost per resolved query and answer quality per route, so the router can be tuned.
The judgement being tested is that cost control should be selective (route by complexity) rather than a blanket reduction that degrades the hard cases where quality matters most.