Generative AI Contact-Centre Assistant: Go/No-Go Against Risk Appetite

Conclusion

The pilot does not support rollout against a low operational risk appetite. The error and disclosure rates are too high, and the control environment has material gaps.

Reasoning

• Factual error 3.1% means roughly 186 of 6,000 drafts had errors. In a full rollout, volume scales, so the absolute number of customers receiving wrong information scales with it.
• Confidentiality breach 0.8% is the most serious: 48 drafts disclosed another customer's information. That is a privacy incident, not a quality issue. A low appetite cannot tolerate a known 0.8% disclosure rate.
• 12% material rewrite means human review is doing substantial work; without it, quality would be unacceptable.
• Prompt injection succeeded in a test. The assistant attempted an API call after an embedded instruction. That is a security control failure, not a model quirk.
• Vendor opacity: no prompt-level logs and silent model updates mean the bank cannot investigate incidents or reproduce behaviour.

Judgement

The pilot measured quality with humans in the loop. Rollout may remove or reduce that review. The relevant question is not how good the model is but what the residual risk is after the controls the bank will actually operate. On current evidence, residual risk is not low.

Assurance value

A SOC 2 Type II report covers the vendor's controls over security and availability for a period. It is useful but limited:

• It does not cover confidentiality here, yet the key risk is disclosure of customer information.
• It does not test the model's behaviour, accuracy, bias or resistance to prompt injection.
• It does not cover the bank's own use of the tool, API permissions, logging or human review.
• Silent model updates mean the vendor's control environment can change after the report period.

Additional assurance needed

1. Model documentation and change notification: contractual right to notice of model updates, version pinning and release notes.
2. Prompt and output logging retained by the bank, with the ability to trace an incident to a model version.
3. Independent red-team testing for prompt injection, data leakage and jailbreaks, including API scope tests.
4. Accuracy and bias testing on the bank's own traffic, with acceptance thresholds.
5. Right to audit or a supplementary control report covering confidentiality.
6. Data processing agreement and confirmation that customer data is not used to train the vendor model.

Judgement

SOC 2 is a starting point, not sufficient evidence for a customer-facing AI system. The bank should map the report's criteria to its own risk appetite and identify gaps explicitly.

Recommendation

No-go for full rollout in six weeks. Approve a limited, controlled pilot extension only if the conditions below are met.

Conditions

1. Human-in-the-loop mandatory for all customer emails until error and disclosure rates fall below agreed thresholds (for example, factual error below 0.5%, disclosure at zero).
2. API least privilege: the balance lookup must be scoped to the authenticated customer, with no ability to retrieve another customer's data. Test this adversarially.
3. Prompt injection controls: input sanitisation, instruction hierarchy, output filtering, and a deny-by-default policy for tool calls.
4. Logging: bank-side retention of prompts, outputs, model version and tool calls; vendor must give change notice.
5. Incident response: a defined path for AI-related privacy incidents, including notification assessment.
6. Board reporting: monthly metrics on error, disclosure, rewrite and injection attempts.

Judgement

A six-week full rollout would scale an unmitigated confidentiality risk. A limited extension with mandatory review and logging generates evidence for a later decision. The board should set explicit exit criteria; if they are not met, stop. The vendor's commercial pressure is not a risk acceptance rationale.