Third-Party Foundation Model With No Training Data Lineage
An insurer licenses a commercial foundation-model API to assist claims triage. The model reads the first-notice-of-loss narrative and proposes a severity band and a reserve estimate; adjusters can override, but an internal audit found they follow the suggestion in 78% of files. Volume is about 340 claims a day. The vendor provides a two-page model card, refuses to disclose training data lineage or evaluation results on the insurer's claim population, describes weights and evaluation methodology as proprietary, and runs inference through a subcontractor in another jurisdiction. The master agreement contains no audit rights, no change-notification obligation, and no right to a fixed model version - the vendor states the endpoint 'improves continuously.' The vendor's SOC 2 report covers infrastructure only. The insurer has no fallback because the legacy rules-based triage engine was decommissioned nine months ago and its rule set was not archived in executable form.
1. How do you validate a model whose internals and training data you cannot inspect? Set out the validation approach and its limits.
Read the worked answer
You cannot validate the model; you can validate the decision system the insurer has built around it. Reframe accordingly and say so explicitly in the validation report, because the distinction is what protects the firm - the residual risk sits in the wrapper, not in the vendor's weights.
Behavioral testing against ground truth. Build a held-out set of 500–800 closed claims with known final severity and reserves. Score them through the live endpoint and measure band-assignment accuracy, calibration of the reserve estimate, and error asymmetry - over-reserving and under-reserving have very different consequences and a single accuracy figure hides that. Because the endpoint changes without notice, this test must be re-run on a fixed cadence with a frozen golden set, and results are only valid as of the test date.
Stability and change detection. The absence of version pinning is itself a control gap. Instrument the endpoint: log every response, run a fixed canary prompt set daily, and alert on output drift. You are substituting continuous output monitoring for the change notification the contract does not give you.
Population fit. The vendor has no evaluation on this insurer's claims. Stratify the golden set by claim type, severity band, and narrative length to find where the model is weakest; the aggregate number will be dominated by high-frequency low-severity claims and will conceal failure in the tail that drives reserve error.
Human-factor validation. The 78% override-following rate means the effective model is the adjuster-plus-suggestion system. Test whether adjusters catch bad suggestions: seed known-wrong suggestions into the workflow and measure detection. If they do not, you have automation bias, and no amount of vendor model quality fixes it.
Limits to state plainly. Behavioral testing validates only the tested distribution; it cannot detect a biased training corpus, cannot verify the subcontractor's data handling, and gives no assurance about the next undisclosed model update.
2. The business wants to sign a three-year renewal as-is. What conditions do you attach, and what is your position if the vendor refuses all of them?
Read the worked answer
Non-negotiable conditions. (1) Fixed-version option with notification: the insurer must be able to pin a model version and receive advance notice of material changes with the option to re-test before adoption. Without this, every validation is stale the day after it is issued and you cannot reconstruct what decided a claim. (2) Audit and assessment rights, including flow-down to the subcontractor and the right to review evaluation methodology - not weights, but how the vendor measures performance. (3) Data handling terms for the other jurisdiction: no retention of claim narratives for training, documented residency, and breach notification inside the insurer's own regulatory clock. (4) Performance commitments on the insurer's population, with a defined remedy, measured by the insurer's golden set rather than the vendor's benchmark. (5) Exit assistance: a documented interface specification and a transition period, since the real exposure is that the legacy engine no longer exists.
If the vendor refuses. Rank the conditions. Version pinning and subcontractor flow-down are existential - walk away without them. Audit rights and benchmark remedies are strongly preferred and can be replaced by enhanced continuous output monitoring, contractual indemnity, and a documented, committee-approved acceptance of the gap. So the answer is not a single yes/no: it is a refusal to sign the package as offered, an offer of a shorter term with the existential conditions, and a board-visible acceptance of the residual items with an owner.
Regardless of outcome, rebuild the fallback. A rules-based triage path, even a crude one, converts the vendor from irreplaceable to replaceable and changes the negotiation. That is the highest-value action available and it does not depend on the vendor at all.