Who Owns the AI Risk When a Vendor Model Sits Behind an Internal Workflow
A hospital system licenses a third-party triage-prioritization model and wraps it in an internal workflow that adds a nurse-review step for the top 10% of scores. The vendor's contract states the model is 'a decision-support tool' and disclaims clinical responsibility; the vendor will not disclose training data composition or feature weights, citing trade secrets. The hospital's AI governance committee was formed six months ago, has a charter, and has approved a risk taxonomy. But the committee has no authority over procurement, and the vendor was selected by the clinical operations team before the committee existed. An internal audit asks: who is accountable if the model systematically under-prioritizes a demographic group? The vendor points to the nurse-review step as the human safeguard; clinical operations points to the vendor's validation report.
1. Allocate accountability among vendor, clinical operations, and the governance committee, and explain the reasoning.
Read the worked answer
Accountability cannot be delegated to a contract clause. The allocation should be:
- The hospital (deployer) is accountable for the deployed system, including the vendor model as embedded in its workflow. Regulatory and ethical frameworks consistently place responsibility for use in context on the deploying organization. The hospital chose to deploy, sets the workflow, and bears the patient impact.
- Clinical operations owns the operational decision to use the tool and the human-review design. It selected the vendor and is responsible for fitness-for-purpose in this clinical setting.
- The governance committee owns the framework and oversight, but its current lack of procurement authority is a structural defect it must remediate, not an excuse.
- The vendor is accountable for the accuracy of its representations and for the model behaving as documented - but 'trade secret' does not discharge the deployer's duty to assess. If the vendor won't provide enough to assess subgroup performance, that is itself a risk finding.
The nurse-review step is a mitigation, not an accountability transfer. A 10% review threshold leaves 90% of decisions effectively automated for prioritization purposes; the safeguard is partial by design.
2. What governance changes would you recommend to close the gap this case exposes?
Read the worked answer
The case exposes a classic 'governance without teeth' pattern: a charter and taxonomy exist, but the committee is bypassed at the point of decision. Recommended changes:
- Gate procurement on governance review. No AI tool enters the workflow without a pre-deployment risk assessment signed by the committee (or its delegate). Retrofit the existing vendor relationship with an emergency review.
- Define a minimum disclosure standard for vendors. Required: validation methodology, subgroup performance where available, known limitations, and change notification. 'Trade secret' can be handled via NDA and on-site review, but refusal to disclose any subgroup evidence should be a documented risk acceptance at executive level, not a silent gap.
- Make human oversight meaningful. Specify what the nurse reviewer sees, what they can override, how overrides are logged, and whether review capacity matches volume. A 10% review rate should be justified against the risk tier.
- Assign a named model owner with authority to suspend use.
- Add a contractual right to audit and a termination/exit plan so the hospital isn't locked in.
- Report to the board on residual risk, especially the undisclosed-subgroup-performance gap.
3. The vendor offers to run a fairness test on the hospital's data but will not share the methodology. Should the hospital accept? What conditions would make acceptance defensible?
Read the worked answer
Acceptance is defensible only with conditions, because an unverifiable fairness claim is weak evidence. The hospital remains accountable regardless of who runs the test.
Conditions that would make it defensible:
- Pre-agreed test specification: which subgroups, which metric(s), what threshold constitutes 'pass', agreed before results are seen.
- Independent verification: hospital or a third party observes the run or reproduces it on held-out data.
- Sufficient output: subgroup-level results with confidence intervals and sample sizes, not a pass/fail summary.
- Documented limitations: known gaps (e.g., small subgroup n, proxy labels) recorded in the risk register.
- Escalation path: a defined response if results are marginal or fail.
- Recorded as residual risk: the committee formally accepts the residual uncertainty and sets a review date.
Without these, the hospital is accepting a vendor assertion as assurance, which is exactly the failure mode the audit flagged.