Hospital AI triage pilot with unresolved safety and data protection gaps
Calderbridge Hospitals Trust runs three emergency departments handling 240,000 attendances a year. It is piloting a vendor's AI triage tool that recommends priority scores from vital signs and free-text notes. The vendor claims 92% agreement with clinicians, based on a 2019 study in another country. Procurement selected the lowest-price compliant bid; the contract has no clinical accuracy warranty and no right to audit the model. A data protection impact assessment is outstanding, and the data processing agreement is unsigned. The tool was introduced in one department after a two-hour webinar; no clinician was consulted on workflow. The board's digital committee was told the tool is 'clinically safe' in a vendor slide. Two near-misses have been logged where the tool under-triaged patients who later deteriorated. The trust's medical director asks the board what governance action is required.
1. Which ISO/IEC 38500 principles are engaged, and where do they conflict?
Read the worked answer
Acquisition is engaged because the trust bought on lowest price without clinical validation, audit rights or warranties. Performance is engaged because the claimed 92% agreement is not evidence of safe performance in this population and workflow. Conformance is engaged because the DPIA is outstanding, the DPA is unsigned, and medical-device and clinical-safety obligations need checking. Human behaviour is engaged because clinicians were given a two-hour webinar and no workflow consultation. Responsibility is engaged because no named clinical owner is accountable for the tool's recommendations. The conflict is real: acquisition pressure favours speed and low cost, while performance and conformance favour delay and evidence. The board should not resolve that conflict by treating all principles as equal. Responsibility and conformance set the limits; within those limits, the board can accept a controlled pilot. A vendor slide is not assurance.
2. What should the board Direct before any wider rollout?
Read the worked answer
The board should direct an immediate pause on wider rollout and a continuation only under a controlled pilot. Direct actions: name the medical director as accountable owner for clinical safety, with a written delegation; require an independent clinical safety case and local validation against the trust's own triage data; require completion of the DPIA and signature of the DPA before any further processing; require contractual rights to audit the model and to receive performance data; require clinician co-design of workflow and training before go-live; and set a near-miss reporting route that does not punish staff for overriding the tool. The board should also require a rollback plan. The human behaviour principle matters here: clinicians must be able to override the tool without friction, and accountability for triage must remain with the clinician. If the vendor refuses audit rights, that is evidence for stopping, not a procurement detail.
3. How should the board Monitor the pilot without managing it?
Read the worked answer
Monitor should test whether the pilot is safe and whether the controls are working. Metrics should include clinician override rate and reasons; under-triage and over-triage rates against clinical outcome; time from arrival to treatment; near-miss and incident reports; completion of training; data protection incidents; and vendor model-update notifications. The board should receive a monthly one-page report from the accountable medical director and an independent review at the end of the pilot. Escalation triggers should include any serious harm or near-miss linked to the tool, two consecutive months of adverse under-triage, missing DPIA actions, or a vendor model change without notice. The board should not monitor by asking 'is the AI accurate?' It should ask whether the sociotechnical system - tool, workflow, training and override - is producing safe decisions. That keeps the board in Monitor rather than in operational management.