The Chatbot That Answered Too Well
A mid-size insurer launches an internal 'ask anything' assistant over its policy documents to help claims adjusters. Adoption is strong. Six weeks in, the security team finds that the assistant will answer questions like 'what is the reserve threshold above which a claim needs manager sign-off' and 'summarize the fraud indicators we use' - accurately. These answers come from documents that were correctly permissioned in the source system, but the assistant's retrieval index was built by a contractor who flattened all documents into one store with no per-user access control. The assistant also confidently answers questions about claims it has no data for, inventing plausible reserve figures. The insurer is regulated and must demonstrate access controls to auditors.
1. Explain the two distinct failure modes here and why 'the documents were correctly permissioned' is not a defense.
Read the worked answer
Failure mode 1 - Broken access control (over-retrieval). The source documents had correct permissions, but the retrieval index did not carry them. When the contractor flattened everything into one store, the assistant lost the ability to enforce 'this user may see this document.' So an adjuster who should not see reserve thresholds or fraud indicators can retrieve them. This is an authorization failure introduced at index-build time, not at the source.
Failure mode 2 - Ungrounded generation (hallucination). For claims with no data, the assistant invents reserve figures. It is generating fluent text with no grounding, and there is no mechanism to say 'I don't have this.' In a regulated claims context, a plausible-but-fabricated reserve number is worse than 'no answer,' because an adjuster may act on it.
Why 'correctly permissioned' is not a defense: permissions in the source system are meaningless if the retrieval layer ignores them. The assistant is a new access path to the same data, and it must inherit and enforce the same controls. Auditors evaluate what the system actually enforces, not what the source system intended. Flattening permissions is a control failure regardless of source hygiene.
2. The insurer must keep the assistant (adoption is strong) and satisfy auditors. Propose the remediation in priority order, and state what evidence you would give auditors.
Read the worked answer
Remediation, priority order:
- Restore per-user access control at the retrieval layer (P0). Rebuild or wrap the index so every chunk carries its source ACL and retrieval filters by the requesting user's entitlements. Until this is done, restrict the assistant to non-sensitive document sets or disable it for sensitive queries. This is the auditor-blocking issue.
- Ground and refuse (P0/P1). Require answers to cite the source passage, and make the assistant return 'no source found' rather than generate when retrieval is empty. For numeric questions (reserves), do not let the model compute or recall - surface the source figure or nothing. Add a confidence/abstention path.
- Separate sensitive content (P1). Move reserve thresholds and fraud indicators into a restricted collection with explicit allow-listing, so even a retrieval bug can't surface them broadly.
- Logging and monitoring (P1). Log queries, retrieved doc IDs, and refusals; alert on access attempts outside a user's entitlement. This turns a silent failure into a detectable one.
- Vendor/contractor accountability (P2). Add an acceptance test for access control to the contractor's deliverable and to future index builds.
Evidence for auditors:
- A test record showing a user without entitlement is refused (and one with entitlement succeeds) - reproducible access-control tests.
- The ACL data-flow diagram: source permissions → index metadata → query-time filter.
- Sample logs of queries, retrievals, and refusals.
- The abstention/citation behavior demonstrated on a no-data claim.
- A change record showing the contractor's flattened index was replaced and re-tested.