GenAI Assistant Expansion and Hallucination Risk
A retail bank deploys a vendor-hosted large language model assistant for 1,200 contact-center agents. The assistant answers questions about fees, disputes, and account closures. Offline evaluation shows a 4% hallucination rate overall, but 18% on fee-waiver questions. A red-team test finds that PII redaction fails in 0.7% of prompts, and the vendor retains logs for 30 days in another country. The vendor provides no model card and will not allow fine-tuning. The bank wants to expand the assistant to customer-facing chat within 60 days. Current controls include an agent review step for agent-facing use, but no real-time guardrails for customer chat. The model risk committee must decide on expansion and set conditions. The bank is subject to model risk management and privacy requirements.
1. Assess the risk and recommend a risk tier.
Read the worked answer
The assistant should be high-risk or tier 1 for customer-facing use because errors affect fees, disputes, and account access, and PII is involved. The current 4% hallucination rate is already too high for unassisted customer use; 18% on fee waivers is a material financial and conduct risk. PII redaction failure at 0.7% of prompts is a privacy incident risk, and 30-day cross-border log retention adds legal and vendor risk. The vendor's refusal to provide a model card or allow fine-tuning limits validation and remediation.
Agent-facing use with mandatory review is lower risk but not zero. Expansion to customer chat within 60 days is not supportable under current controls. The committee should treat this as a material change requiring revalidation, not a simple channel extension.
2. What controls must be in place before expansion?
Read the worked answer
Before any customer-facing expansion:
- implement retrieval-augmented generation grounded in approved policy documents;
- add real-time guardrails for PII, prompt injection, and prohibited advice;
- require human escalation for fee waivers, disputes, and account closure;
- add clear disclosure that the user is interacting with an AI and how to reach a human.
Fix redaction and test it on adversarial prompts. Negotiate a model card, audit rights, data residency, and log-retention terms; if unavailable, use a different vendor or self-hosted model. Run a limited pilot with a kill switch, daily review of transcripts, and a rollback plan. Set go/no-go criteria: hallucination rate below 1% on a golden set, PII leakage below 0.01%, and 100% escalation for high-risk intents.
3. How should the bank evaluate and monitor the assistant after launch?
Read the worked answer
Evaluation should combine offline golden sets, red-team adversarial tests, and online monitoring. Track hallucination rate by intent, PII leakage, escalation accuracy, containment rate, customer complaint rate, and policy-violation rate. Use human review sampling and automated checks.
Monitor vendor changes and model updates as change events. Define incident response for privacy leakage and harmful advice. Report monthly to model risk and compliance. Revalidate before any new intent or model version. Keep agent-facing and customer-facing evaluations separate; do not assume agent-facing performance transfers.