The Model That Learned Too Much
A bank deploys an internal AI assistant trained on 10 years of customer service transcripts to help agents draft replies. Before deployment, the team removed names, account numbers, and phone numbers from the training data. A red-team exercise finds that prompting the assistant with "summarize the last conversation about the Henderson mortgage" returns a detailed summary including the customer's address and the amount of a disputed charge - details that were not in the redaction list. The assistant also, when asked "what do customers like me usually get offered," produces a response that closely tracks a specific customer's complaint. The bank's model risk policy requires an annual review and says "models must not reproduce customer personal data." Deployment is scheduled in two weeks. The vendor's contract disclaims liability for "outputs generated by the model."
1. Diagnose the failure. Why did redacting names, account numbers, and phone numbers not prevent reproduction of personal data?
Read the worked answer
Because redaction targeted direct identifiers and the model memorized quasi-identifiers and context.
- Addresses and disputed-charge amounts were never on the redaction list. They are personal data, and in combination they are highly identifying - an address plus a charge amount plus a mortgage reference can point to one household.
- Context is a quasi-identifier. "The Henderson mortgage" is a name, but even with names removed, a conversation about a specific mortgage dispute is unique enough that the model can retrieve and reconstruct it. Memorization is about the uniqueness of the record, not just the presence of a name field.
- The "customers like me" output shows the model is not just retrieving but generalizing from small clusters - which means it can leak one person's complaint as a "typical" pattern.
The core error: the team treated redaction as a data-cleaning task when it is a risk question. The right question is not "did we remove the obvious identifiers?" but "can any training record be reconstructed or inferred from an output?" Redaction of a fixed field list cannot answer that.
2. The policy says models must not reproduce customer personal data. Is the model out of policy, and can it be deployed in two weeks?
Read the worked answer
It is out of policy. The red-team output reproduces an address and a disputed-charge amount tied to an identifiable customer - that is customer personal data, reproduced. The policy does not say "must not reproduce unless the identifiers were redacted"; it says must not reproduce. The model fails as demonstrated.
Can it deploy in two weeks? Only if the reproduction is eliminated and verified, not mitigated by a warning banner. Options, in order of strength:
- Retrain or fine-tune with stronger protections: differential privacy, deduplication of near-unique records, and removal of free-text fields that carry quasi-identifiers. This is the durable fix but unlikely in two weeks.
- Restrict retrieval: if the assistant is retrieval-augmented, gate retrieval to the current customer's own records only, so it cannot surface another customer's conversation. This is achievable quickly and addresses the Henderson failure directly.
- Output filtering: block outputs containing address patterns or amounts not belonging to the active customer. Necessary but not sufficient - filters miss paraphrase.
Recommendation: do not deploy on the current design. Deploy only with retrieval scoping (option 2) plus output filtering, and re-run the red team against the scoped design. If that cannot be verified in two weeks, delay. The policy is a gate, not a suggestion.
3. The vendor disclaims liability for model outputs. What does that mean for the bank's risk position, and what should change?
Read the worked answer
It means the bank owns the output risk entirely. A disclaimer for "outputs generated by the model" shifts the consequence of exactly the failure that occurred - reproduction of personal data - onto the deployer. The bank is the controller of the customer data and the operator of the assistant; it cannot contract away its regulatory obligations, and the disclaimer ensures it also cannot recover from the vendor.
What should change:
- Do not accept a blanket output disclaimer for a model processing customer personal data. Negotiate: vendor liability for failures of the model to meet agreed privacy specifications, and indemnity for third-party claims arising from model reproduction.
- Require disclosure of training data provenance and any memorization testing the vendor performed - the bank cannot assess reproduction risk it cannot see.
- Contract the verification: a right to red-team and to receive results, plus notification of known memorization issues.
- Internal ownership: name a model owner accountable for the annual review, and make the red-team result a documented finding that must be closed before deployment.
The judgement: a vendor disclaimer does not reduce the bank's exposure - it only removes the bank's recourse. Price that into the decision to deploy at all.