Ada at the Contact Center: Credit Reports in the Prompt Window
Cornerstone Bank has deployed "Ada," a vendor-hosted LLM assistant, to 900 contact-center agents. Agents paste customer context into the prompt window: account numbers, transaction histories, and occasionally full credit reports. The vendor's order form grants Cornerstone a license to "Customer Content" and permits the vendor to "use Customer Content to provide, maintain, and improve the Services," with 30-day retention for abuse monitoring and no stated training exclusion. No DPIA was completed; procurement routed the purchase as a $180,000 SaaS renewal under an existing vendor's master agreement.
Two incidents in the first month. An agent pasted a customer's entire credit report, including a bankruptcy filing, to draft an empathetic response; the customer later complained. Separately, Ada told a customer - in a chat the agent copied verbatim - that a $35 late fee would be waived. The fee was not waived, and the customer has screenshots and is asking for the waiver plus compensation. Cornerstone's model risk policy classifies any AI system that "influences a customer outcome" as Tier 2, requiring pre-deployment review, which never happened.
1. Identify the failure points in this deployment and rank them by severity, with reasoning.
Read the worked answer
Rank by reversibility and scope - what cannot be undone, and how many people it touches.
- Customer data flowing into a vendor that may train on it (highest). Full credit reports and account numbers entered a system whose terms permit use to "improve the Services," with no training exclusion and 30-day retention. This is a confidentiality failure with a legal tail: GLBA safeguard obligations, GDPR Art. 28 processor terms, and a disclosure to customers that likely contradicts the privacy notice. It is irreversible - you cannot recall data already ingested - and unbounded: 900 agents over a month is tens of thousands of prompts.
- No DPIA and no Tier 2 review. This is the enabling failure. Every other item was preventable by the review that policy required and procurement skipped. Its severity is high precisely because it is systemic: the next AI purchase will skip the same gate unless the routing is fixed.
- No input controls. Free-text paste of a credit report is a design flaw, not an agent error. There is no DLP, no redaction, no field allowlist.
- Hallucinated customer commitments. Lower volume, but it converts a model error into a consumer-conduct event, and repetition turns it into a pattern regulators will ask about.
- Procurement misclassification. A $180k renewal that changed the nature of the service should have triggered new vendor due diligence. Process defect with a clean fix.
Why this order: items 1 and 3 are one risk seen from both ends - data leaving the boundary. Item 2 is why nobody saw either. Item 4 is merely the symptom that made the rest visible.
2. What must be true before Ada stays in production?
Read the worked answer
Three gates: contract, controls, accountability.
Contract:
- An explicit no-training clause on Customer Content, with technical attestation rather than a promise.
- Retention limit (e.g., seven days for abuse monitoring), deletion on request, subprocessor list with change notice.
- Breach notification inside 24 hours, audit rights, and a security exhibit that is not overwritten by the master agreement.
Controls:
- Redaction at the edge: a deterministic scrubber for account numbers, SSNs, and credit-report field patterns before any prompt leaves the bank; block paste of documents above a size threshold.
- Constrain Ada to allowlisted task templates. No free-form "paste the customer's file here."
- Logging that records what was sent, so the scope of any incident can be measured rather than estimated.
Accountability:
- A retroactive DPIA and Tier 2 review signed by a named model owner, before the next customer interaction.
- Human confirmation required for any commitment about fees, rates, or timelines. Ada drafts; the agent commits.
- Weekly transcript sampling for hallucination plus a complaint tag, so incidents are counted rather than discovered.
The operational test: if the bank cannot answer "what customer data did we send, to whom, and what did they do with it" for the past 30 days, it is not ready. Right now it cannot.
3. How should Cornerstone respond to the customer holding the $35 fee-waiver screenshot?
Read the worked answer
Honor the waiver, compensate, then determine whether she is the first case or the fiftieth.
- Honor it. An agent relayed a commitment in the bank's voice. Whether a model or a person generated the sentence is an internal fact that does not change the customer's reasonable reliance. Contesting $35 on the ground that "the AI got it wrong" spends real credibility to save a trivial sum, and it tells every customer who hears the story that the bank litigates its own errors.
- Compensate for the friction - a modest credit beyond the $35, same-day resolution.
- Root-cause, do not scapegoat. If the agent is disciplined for using the tool the bank provided, the next month of incidents goes unreported. Fix the tool and the policy that allowed commitments to be made in chat.
- Sample the logs. One screenshot implies a base rate. Search 30 days of transcripts for commitment language ("waived," "refunded," "approved") and reconcile against the core system. If there are more, proactive remediation is cheaper and more ethical than waiting for each customer to complain.
- Disclose. Update the customer-facing AI disclosure so customers know when they are dealing with a model-assisted agent.
The principle: an organization cannot deploy a system to speak for it and then disown what it says. Accountability means the error belongs to the bank, the remedy belongs to the customer, and the correction belongs to the system.