Nine Tier-1 Models, Two Breaches, One Analyst: Rebuilding an AI Risk Appetite

Assessment: the statement is a slogan with numbers attached. It meets one test - it is measurable - and fails the others: 'outside tolerance' is undefined per model, no owner is named, and there is no escalation ladder. The prior quarter's pack says 'within appetite' while 22% of tier-1 models breach; the statement and the reporting already contradict each other, which is the most damaging finding because it destroys the credibility of the whole framework.

Redesign

• Define tolerance per model by tier: e.g. AUC below a stated floor, PSI above a stated value, calibration error above a ceiling, override rate above a threshold. Tolerance must be a number agreed with the model owner, not a feeling.
• Keep the 5% aggregate limit but add a per-model rule: no tier-1 model remains out of tolerance for more than one quarter without an approved remediation plan.

Concentration Count economic dependency, not contracts. One cloud provider plus its foundation-model partner plus a shared fine-tuning pipeline is one correlated failure, not three vendors. On that basis 4 of 9 (44%) already breaches the 30% limit.

KRI set (report monthly, escalate on breach)

• % tier-1 models validated on schedule; % overdue.
• Number of tier-1 models out of tolerance; mean days to remediate.
• Drift alerts open beyond 30 days.
• Third-party change notifications received versus changes detected.
• Human override rate by model; complaint and incident counts.
• Shadow AI instances discovered.

Aggregation honesty With one FTE and no tooling, full quantitative aggregation is not achievable. Use a tier-weighted heat aggregation by business line and state the limitation explicitly rather than implying precision you do not have.

Appetite is breached, not 'within appetite'. 2 of 9 tier-1 models is 22% against a 5% limit. Say that plainly; the credibility cost of spin exceeds the cost of the breach.

Escalation CRO notifies the CEO and board risk committee within the framework's stated window, with a remediation plan and a residual-risk statement. Assess whether supervisory notification is required - in insurance, a claims-triage failure can affect policyholders directly and may be material; assess, do not assume either way.

Prioritisation Rank by exposure and detectability, not by ease of fix. Claims triage affects claimants and can produce adverse decisions, complaints and regulatory interest - it outranks marketing or service models. Both breaches sit in the same function, which points to a shared cause: a vendor update, a change in claims mix, or a common feature source. Investigate jointly before funding separate fixes; two independent remediations may be wasted spend.

Sequence given no tooling budget

1. Immediate compensating control - human review of affected claims decisions.
2. Root-cause analysis on the shared-cause hypothesis.
3. Recalibration or re-validation, whichever the evidence supports.
4. Vendor escalation if the cause is external, with contractual leverage.

Set a decision date and a decommission criterion. If the model cannot return to tolerance within one quarter, retire it. Report the breach once, with a plan and a date - repeatedly listing it as a 'known issue' is how appetite statements die quietly, and internal audit will find it next cycle.

Remove

• The raw model count - a number with no denominator, no trend and no risk content.
• Training-session counts - activity metrics that measure effort, not exposure.
• The unsupported sentence 'AI risk is within appetite'. Never assert appetite status without a scorecard behind it.

Add

• Appetite scorecard: one row per limit, green/amber/red, current value, trend, and an explicit statement of whether appetite is breached.
• Tier-1 model page: tolerance status, trend, owner and remediation date per model.
• Top five risks with owner, mitigation and target date.
• Incidents in the quarter with impact and lesson - not a list of tickets.
• Concentration map: vendor and foundation-model dependency mapped to models and business lines, including correlated failure paths.
• Validation coverage: validated, due, overdue.
• Forward look: one or two items - an upcoming foundation-model release, a regulatory change, a supplier renewal.

Make it decision-useful

• Show trends, not snapshots. A board needs direction of travel, not a still image.
• State what is not known: the 61-model register is probably incomplete and depends on one analyst - a single point of failure the board should see.
• Close with two or three explicit decisions requested, each with a recommendation.

Ten-week plan Weeks 1–3: agree tolerance definitions and build the scorecard. Weeks 4–6: populate and reconcile with the register; run the correlation analysis audit asked for. Weeks 7–8: draft narrative and dry-read with one non-executive director. Weeks 9–10: CRO review and issue. Use the additional FTE for data reconciliation, not for writing slides - the analyst is the constraint.