Regurgitation in a Fine-Tuned Support Model
Larkspur Retail fine-tuned an 8-billion-parameter open-weight model on 14 months of customer-service chat logs - 4.2 million conversations - to power a support assistant in three markets. Redaction before training used regular expressions for email addresses and phone numbers only. A researcher has now produced a real customer's full postal address, card last-four, and order number from the production assistant; 11 of 500 sampled prompts returned verbatim identifiers. The training data sits in a data lake readable by 60 engineers, and retention policy says 24 months although the lake holds 38 months. No DPIA was completed. The company has received 1,200 CCPA deletion requests in the past year and has no process for removing data from a trained model. The weights were also copied to a third-party evaluation sandbox last month. The assistant is still live.
1. Set out the first 72 hours: containment, evidence preservation, and the notification analysis.
Read the worked answer
Containment first: suspend customer-facing free-text generation or gate it behind a strict output filter, revoke the sandbox copy of the weights, and stop any further copying. Preserve the model version, training configuration, the redaction script, the researcher's prompts and outputs, and access logs for the lake and sandbox. Sample the model systematically to scope how many distinct individuals are extractable rather than relying on the researcher's 11 hits.
Notification analysis: model outputs that identify a person are personal data, and prompting the model to emit them is unauthorised access, not a curiosity. Under GDPR, notify the supervisory authority within 72 hours of becoming aware if the processing presents a risk to rights and freedoms; under CCPA, notify affected consumers in the most expedient time possible without unreasonable delay if the breach meets the statutory definition; state breach laws vary. The identifiers here - address, order number, card last-four - are enough for convincing social-engineering fraud even without a full card number. Legal should document the risk assessment either way, including a reasoned decision not to notify. The absence of a DPIA before training is a separate compliance failure to remediate now.
2. Explain why the regular-expression redaction failed and design a more defensible de-identification approach, including its limits.
Read the worked answer
Regex succeeds only on fixed patterns. Names, postal addresses, order numbers, and identifiers embedded in free text or split across conversational turns pass straight through, and no recall measurement was performed, so the team had no idea what fraction was missed. Fine-tuning on 4.2 million conversations gives the model many opportunities to memorise rare strings, which are exactly the ones a regex cannot recognise.
A defensible approach starts with minimisation: do not fine-tune on raw logs at all if the task does not require it. Where it does, use named-entity recognition plus human review on a statistically valid sample, entity-level tokenisation with consistent surrogates, removal or deduplication of rare strings, and extraction-attack testing with injected canary identifiers to measure leakage before release. Differential privacy during training reduces memorisation at a utility cost.
Limits must be stated: no de-identification is perfect, NER recall is never 100%, membership inference remains possible, and utility degrades. Treat the model as holding personal data unless you can evidence otherwise, and re-test after every retrain.
3. How should Larkspur handle the 1,200 deletion requests against a trained model, and what should it tell the regulator?
Read the worked answer
Exact deletion from weights is not reliable today. Options, in order of strength: retrain from a filtered dataset excluding the requesters' data; apply an approximate unlearning method with empirical verification that the individual's information can no longer be extracted; and, as a compensating control only, block known identifiers at inference. Blocklisting is not deletion and should be labelled as such.
Operationally, map the 1,200 requesters to training records, check whether their data is in the model's memorised set, and prioritise anyone whose identifiers are extractable. Verify by running extraction attempts against the retrained or unlearned model. Document the method, the verification result, and the residual risk for each request.
To the regulator: describe the incident, the scope, the containment, the lawful basis and retention position, the fact that a DPIA is now being completed, and a dated remediation plan. Avoid claiming the training data was 'anonymised' - the extraction result contradicts that, and an unsupportable claim is worse than the original failure. Provide the verification evidence and commit to ongoing extraction testing.