Expanding an AI Assistant After It Surfaces Restricted Content
Harlow Mutual is a 3,500-person insurer operating in the UK and Canada. It is piloting AskHarlow, an AI assistant that indexes email, SharePoint, claims notes, and HR files and answers natural-language questions. The 300-user pilot shows strong adoption and useful answers. Legal review finds the assistant has quoted a draft legal opinion, an HR investigation memo, and a claims note containing medical details. The records team reports that retention labels are missing on 60% of repositories, and litigation holds apply to two active matters. The vendor wants to expand to all 3,500 staff in 60 days. The CISO wants assurance that permissions, cross-border transfer, and deletion are controlled. The KM lead chairs the governance group.
1. What governance failures does the pilot reveal? Distinguish content, access, and records failures.
Read the worked answer
The pilot reveals three linked failures.
Content failure: draft legal advice and an HR investigation memo are not authoritative general-corpus content. Indexing them without purpose limitation turns a working draft or an investigation record into an answer source. The assistant is not a system of record and should not become a shadow repository.
Access failure: the assistant quoted material that should have been restricted to legal, HR, or claims handlers. Either permissions were not enforced at retrieval or the index inherited over-permissive access. Permission trimming must happen at index and query time, not only in the user interface.
Records failure: missing retention labels on 60% of repositories and active litigation holds mean the assistant cannot reliably distinguish preserve, restrict, and dispose. Cross-border processing between the UK and Canada adds transfer and localization obligations.
There is also a purpose-limitation issue: HR investigation content was used for a general Q&A purpose. The governance group should treat this as a data protection and records issue, not only a search relevance issue.
2. Recommend a go/no-go decision and the controls required before expansion.
Read the worked answer
No-go on the vendor's 60-day full expansion. Approve a conditional expansion to a bounded cohort only after controls are tested.
Required controls:
- Permission-aware retrieval with index-time and query-time enforcement, tested against HR, legal, and claims scenarios.
- Exclusion rules for draft legal advice, HR investigations, and matters under litigation hold.
- Sensitivity labels and DLP for personal, medical, and privileged content.
- A data map covering UK and Canada repositories, with a transfer impact assessment.
- Retention labels applied to high-risk repositories before indexing; legal-hold flags respected.
- Deletion propagation from source to index and logs within a defined SLA.
- Audit logging, user feedback, incident response, and acceptable-use training.
- Human review for high-risk uses such as claims, HR, and legal.
Assign accountable owners: CISO for access, General Counsel for privilege and holds, Records for retention, and KM for content quality. Set exit criteria and re-test before broad rollout. Expansion is a governance decision, not a procurement milestone.
3. How should retention and deletion work when the assistant is not the system of record? What about a request to delete personal data?
Read the worked answer
The assistant is a derived index. Retention and disposition must be controlled at the source system, and the index must inherit the source's labels, holds, and permissions. Do not create a separate retention schedule for the assistant.
Operationally:
- Map every indexed item to its source record and owner.
- Respect legal hold: held content is preserved and excluded from general answers where privilege or investigation sensitivity applies.
- When a source item is deleted under an approved disposition rule, delete the derived index entry and associated cached content within a defined SLA.
- Log deletions and test them; an untested deletion path is not a control.
For an erasure request, search the index, caches, and logs as well as the source. Apply legal and regulatory retention exceptions, including litigation hold and insurance recordkeeping requirements. If an exception applies, restrict processing rather than deleting. If no exception applies, delete or anonymize the derived data and confirm to the requester. The governance group should approve the procedure and the Records team should own the schedule.