Bank Assistant: Over-Refusal, Fabricated Policy, and a Request to Automate Credit Decisions

Treat refusals as a measurable defect with two error rates, not a single switch.

• Build both eval sets. A benign set of 200-300 real officer requests, deliberately including the words that trigger refusals (fraud, declined, denial, risk), and an adversarial set covering prompt injection, attempts to extract other customers' data, and requests for advice outside the assistant's scope.
• Tune per category, not globally. Configure safety thresholds per harm category in the platform. Categories driving the false refusals on internal risk vocabulary can be relaxed for this deployment; categories covering dangerous content, self-harm, and data exfiltration stay strict. Relaxing everything is how the second incident happens.
• Add scope clarity. A system instruction defining the assistant as an internal credit-risk tool, authorized to quote internal policy and discuss declines, resolves a large share of refusals cheaply - but verify on the eval set rather than assuming.
• Route the middle. Borderline requests should return a clear 'escalate to a human underwriter' path instead of a flat refusal, which is better for the officer and safer for the bank.
• Re-run both sets after every model, filter, or prompt change and track false positive and false negative rates side by side. Also fix the fabricated policy number with grounding: retrieval over the actual policy library, citation required, and 'policy not found' as an allowed answer.

Recommend declining the proposal as specified.

The problems are structural, not fixable with a better prompt:

• Explainability: adverse action requires specific, articulable reasons. A generative model's output is not a stable decision rule, and 'the model said so' does not satisfy a regulator or an appeals process.
• Reproducibility: generation is non-deterministic across model versions; the same application could yield different outcomes, which is indefensible in credit decisions.
• Fair lending: disparate impact testing requires a defined feature set and a model that can be evaluated across protected groups. You cannot run that analysis meaningfully on an unconstrained language model.
• Accountability: no human is in the loop, so there is no owner of the decision.

What is defensible instead: keep the assistant in an assistive role - extract and summarize financials, flag missing documents, draft the credit memo, surface relevant policy - with a human making and signing the decision. If the bank genuinely wants automated small-ticket approvals, that is a governed statistical or machine-learning credit model with defined features, fair-lending validation, explainability tooling, and an appeals path. The generative assistant can support that pipeline; it should not be the decision-maker inside it.

• Logging and retention: prompts, retrieved sources, and outputs logged with access controls and a defined retention period; immutable enough for audit sampling, restricted enough that the log is not itself a data risk.
• Quality monitoring: sampled human review on a fixed cadence; automated groundedness checks confirming that cited policy passages exist and support the claim; refusal-rate tracking with a threshold that triggers investigation.
• Drift detection: monitor input distribution, refusal rate, retrieval hit rate, and tool-call failure rate against launch baselines. A new product or policy changes the question mix long before anyone files a ticket.
• Fairness review: where outputs touch customer outcomes, disaggregate quality and refusal metrics across applicant segments and report quarterly - even for an assistive tool, because a summarizer that omits relevant income for one segment is a fair-lending problem.
• Adversarial testing: quarterly red team, plus a regression suite seeded with every production incident.
• Change control: documented model, prompt, and retrieval changes with a rollback path; model cards and a change log available to Compliance; a named owner for the assistant and an incident escalation route.
• Human-in-the-loop assurance: a written statement of which decisions the tool may inform and which it may never make, with a technical control enforcing it where possible.