The LLM Loan Memo Assistant
A $4B credit union pilots a third-party LLM assistant for commercial loan officers. The assistant summarizes applicant financials, drafts credit memos, and flags 'missing documents.' It runs on a vendor API with a 90-day data retention clause and no fine-tuning on bank data. In a 300-application pilot, loan officers accepted 94% of drafts with light edits; in 6 cases the assistant invented a debt obligation that did not exist, and in 2 cases it omitted a delinquent account that appeared in a PDF table. The vendor's contract caps liability at $50k and disclaims all warranties. The bank's model risk policy requires 'effective challenge' and annual validation for models materially affecting credit decisions.
1. Is the assistant within scope of the bank's model risk policy? Classify its risk and justify.
Read the worked answer
In scope. The policy's trigger is whether a model materially affects a credit decision, not whether the model is a traditional scorecard or whether a human signs the final memo. The assistant's output - summarized financials, flagged missing documents, and drafted credit rationale - feeds directly into underwriting. A 94% acceptance rate with light edits shows the output is doing real work in the decision process.
Risk classification: high. Reasons:
- Consumer/commercial credit impact: errors can lead to wrongful approval or denial.
- Generative model uncertainty: hallucinated obligations and omitted delinquencies are exactly the failure modes that credit review is meant to catch.
- Third-party concentration and opacity: the bank does not control the model, cannot inspect weights, and has a 90-day retention clause.
- Overreliance: high acceptance rates suggest automation bias.
- Contractual mismatch: a $50k liability cap is trivial relative to credit losses.
The 'productivity tool' label does not exempt it. If the bank treats it as a word processor, it will skip effective challenge, validation, and monitoring - precisely the controls the evidence shows are needed.
2. Identify the top five risks and the controls you would require before scaling, including third-party risk.
Read the worked answer
Top risks and controls:
- Hallucinated facts (invented debt). Control: constrain the assistant to extractive summarization with citations to source pages; prohibit free-form assertions of fact unless linked to a document; require loan-officer attestation on each cited figure.
- Omissions (missing delinquent account). Control: use deterministic extraction for tabular financials; run a completeness check against a document checklist; sample-audit memos against source files.
- Overreliance / automation bias. Control: display a 'verify each figure' prompt; require a second reviewer for memos above a threshold; track edit rates and override reasons.
- Data leakage and retention. Control: negotiate no-training-on-bank-data, shorter retention, regional processing, and deletion certification; use a private instance if available; redact unnecessary PII.
- Third-party opacity and liability. Control: require model documentation, evaluation reports, change notices, audit rights, security attestations, and a liability cap proportionate to risk; add termination and transition rights.
Cross-cutting control: build a labeled evaluation set of at least several hundred applications, including adversarial PDFs and tables, and measure precision/recall for 'missing document' flags and factual accuracy before scaling.
3. Make a go/no-go recommendation and state what evidence would change it.
Read the worked answer
Recommendation: no-go for production expansion. Continue only as a tightly controlled assistive pilot, with loan officers required to verify every factual assertion against source documents and with 100% review of AI-drafted memos for the first 60 days.
Rationale: the observed error rate is small in percentage terms but unacceptable in credit: 6 invented obligations and 2 omitted delinquencies in 300 applications is a material defect, and the contract allocates almost none of the consequence to the vendor. The high acceptance rate compounds the risk.
Evidence that would change the decision:
- A pre-registered evaluation on at least 500 local applications showing zero unsupported factual assertions and a defined recall floor for delinquencies.
- Demonstrated citation accuracy: every generated figure traceable to a source page.
- Vendor agreement to no-training, shorter retention, audit rights, and a liability cap that reflects potential credit loss.
- Monitoring data showing loan officers actually verify citations, not just accept drafts.
- A documented effective-challenge review by a function independent of the pilot team.
If the vendor will not provide documentation or contract changes, replace the tool. The bank can accept a productivity loss; it cannot accept a credit decision based on a fabricated debt.