Calder Mutual: Defining Scope and Independence Across a Three-Lines-of-Defense AI Estate

Define scope by decision impact, not by algorithm or by the label 'AI'. A workable definition: any model, rule set, LLM application or third-party score that materially shapes an outcome for a customer, a claim, a price or a regulatory report, and whose logic is derived from data or maintained as logic over time, is in scope. Depth of control then varies by tier.

Applying that to the five:

• Pricing model: in scope, top tier. Directly prices customers and is likely to attract regulatory attention in both underwriting and fairness.
• Claims summariser: in scope. It does not set the reserve, but it shapes the adjuster's decision while appearing authoritative; hallucination risk and PII handling in claim narratives are the core risks.
• Agent chatbot: in scope. Retrieval grounding, prompt injection and disclosure that the user is talking to a machine.
• Rules engine: in scope. 'Not AI' is the wrong test. It is a model by behaviour: 41 changes a year with no recorded rationale or testing is exactly the failure mode model governance exists to catch. Its deterministic nature lowers the statistical validation burden, not the change-control and documentation burden.
• Vendor fraud score: in scope. The insurer owns the decision outcome even though it does not own the model. Third-party model risk requires vendor documentation, performance monitoring and contractual rights.

Second implication: an AI program also needs a use-case register, because the chatbot and summariser are applications built on models, not models themselves. The board's 'single view' must join both registers or it will miss half the estate.

Use a small number of tiers and score them on explicit factors: degree of decision autonomy, customer and financial impact, regulatory sensitivity (pricing, claims, vulnerable customers), data sensitivity, and whether the output is generative or opaque.

Tier 1 - material, autonomous or regulated impact. Full validation before deployment and annually, documented effective challenge, formal board or committee reporting, continuous monitoring with defined thresholds, and a named accountable executive.

• GLM-plus-GBM pricing model: regulated pricing, direct financial impact, opacity from the GBM component.

Tier 2 - material but human-in-the-loop. Validation at deployment and on material change, monitoring, documented human oversight design, periodic sample review.

• Claims summariser: material because it nudges reserves, but an adjuster signs; the control that matters is measuring whether adjusters over-rely on it.
• Vendor fraud score: same tier, with the added requirement of vendor documentation and fallback testing.

Tier 3 - lower impact, high change frequency. Change control, testing evidence and periodic sampling rather than full statistical validation.

• Rules engine: not because the impact is low, but because each change is testable; its risk concentrates in release discipline, not in statistical drift. If any single rule materially affects payouts, that rule is lifted to Tier 2.

Chatbot: Tier 2, driven by disclosure and grounding risk.

Why it is defensible: each placement traces to a stated factor, not to opinion. The scheme also resolves the chief actuary's objection without conceding exclusion - the rules engine is governed, just at proportionate depth.

Treat this as an organisational defect first and a tooling question second. Effective challenge requires that the person who built the model does not determine whether it is adequate. That is a three-lines-of-defense principle, not a software feature.

What to do:

1. Name the conflict plainly and quantify it: 210-plus models, two second-line analysts, one builder-validator. The problem is capacity as much as independence.
2. Separate roles in the workflow. The developer owns documentation, assumptions and testing evidence; a second-line validator owns challenge, findings and sign-off. In watsonx.governance these are distinct task assignments and permissions within the model lifecycle, and the audit trail records who did what and when.
3. Provide an escalation route. If second-line capacity cannot cover Tier 1, use external or peer review for the pricing model and reserve internal validation for Tier 2 and 3. That is a defensible portfolio decision.
4. Build capability, not just a rule. Second-line analysts need support to challenge a GBM model; pair them with an independent reviewer for two cycles.

What the platform can and cannot do: it can enforce segregation of duties in the approval workflow, prevent self-approval where configured, and produce the evidence an auditor needs. It cannot create independence, expertise or a mandate over vendor models. Those are decisions for the CRO and the board risk committee, and the implementation should say so rather than let a workflow diagram imply the problem is solved.