The Algorithm in the Ward
You are District Health Officer of a district with a tribal-majority taluk. The state health mission is piloting an artificial intelligence tool that reads chest X-rays to triage presumptive tuberculosis patients. The vendor's brochure cites 96 per cent sensitivity from a validation study on 40,000 images from a foreign hospital. In your nine-week pilot across six primary health centres, the tool flagged 71 per cent of confirmed cases and missed 29 per cent. The missed cases were concentrated among women under 45 with low body mass index, and among patients whose films were taken on the older, lower-resolution machines in two centres. The vendor's contract contains a clause permitting de-identified image sharing for 'product improvement'. A senior officer in the mission wants a state-wide rollout before the annual TB review. A newspaper has already reported the pilot as a success.
1. Assess the technical and ethical problems with the pilot and with the contract.
Read the worked answer
Technical.
- External validity. 96 per cent on 40,000 foreign, hospital-grade images does not transfer to six primary health centres with older machines, different exposure protocols and a different case mix. The number that governs here is 71 per cent.
- Differential failure. Missing 29 per cent overall is serious. Missing them disproportionately among women under 45 with low BMI is worse, because that is exactly the group in which TB is already under-diagnosed clinically. The tool is reproducing a bias in its training data, not correcting one.
- Reference standard. How were 'confirmed cases' defined? Without a stated reference standard, the sensitivity estimate is not reproducible and cannot be compared with the vendor's claim.
- Use as a rule-out. A triage tool that misses three in ten is an adjunct at best. It is unsafe as a filter that sends negatives home.
Ethical and governance.
- Data. 'De-identified image sharing for product improvement' is a perpetual, global, uncompensated transfer of patients' health data, with real re-identification risk from imaging metadata and rare presentations. Consent was not taken for that purpose. Under the Digital Personal Data Protection Act, 2023, the district needs a lawful basis it currently does not have.
- Precaution. A rollout before local validation is experimentation on the public without ethics review or an opt-out.
- Transparency. The newspaper has already reported a success. The record is wrong and correcting it is part of the duty.
- Accountability. Procuring an unvalidated tool may be naivety or may be capture; either way it carries your signature.
The core judgement. A tool that is 96 per cent sensitive in one place and 71 per cent in another is not one tool. Deploying it without local validation and a fallback clinical pathway turns a statistic into missed patients.
2. What will you do before any scale-up? Design the evidence you need and the contract changes you will demand.
Read the worked answer
Evidence - design it before you argue about it.
- A prospective evaluation on roughly 1,200-1,500 presumptive patients, stratified across urban, rural and tribal facilities, with a pre-registered protocol.
- A single reference standard (CBNAAT or Ultra) applied to all participants, and readers blinded to the AI output.
- Pre-specified subgroup analysis by sex, age, BMI, facility and machine type, with a pre-agreed stopping rule: if sensitivity in any subgroup falls below a stated threshold, the tool is not deployed as a rule-out in that subgroup.
- A human-in-the-loop pathway: the AI cannot be the sole arbiter of a negative; every negative gets clinical review and every patient gets follow-up.
- Cost per additional case detected, compared with the existing microscopy and CBNAAT pathway - because a tool that finds fewer cases at higher cost is not innovation, it is procurement.
Contract changes.
- Remove or narrow the data-sharing clause: model improvement on-site or federated, no export, a data protection impact assessment, and explicit patient information.
- Performance warranties tied to local validation, not brochure claims, with a termination right without penalty if subgroup thresholds fail.
- No exclusivity and an open interface, so the district is not locked in for a decade.
Process. Put the scale-up decision in writing to the mission director, setting out the evidence gap and proposing a twelve-week window with a defined end point. Do not simply refuse - a refusal without an alternative reads as obstruction and will be overridden. Offer the evaluation as the route to scale-up, so that the officer who wants speed gets a date rather than a denial.
3. The newspaper has already called the pilot a success and the mission wants to move. How do you correct the public record without undermining confidence in the health system?
Read the worked answer
Principles: correct the fact, not the institution; be first, be specific, be repeatable.
- Within 48 hours, issue a short factual statement: the pilot is ongoing; the 96 per cent figure is the vendor's external validation, not a district result; the district's own evaluation is in progress and will be published; no scale-up decision has been taken.
- Brief the frontline first - medical officers, laboratory technicians, ASHA and ANM staff - before the press. If they learn it from the newspaper, the correction has already failed.
- Give the reporter the same numbers you have, with context. A journalist corrected privately writes a better second story than one contradicted publicly, and the second story is the one that reaches patients.
- The part that actually protects patients: a one-page note in Kannada and in the local language of the tribal taluk, at every facility, stating what the tool does, that a doctor still decides, and that anyone whose symptoms persist must return for review. A patient who trusts an AI negative and does not come back is the real harm - bigger than any headline.
- Record the correction in the file, so the next officer inherits the true version rather than the newspaper's.
What I would not do. Issue a vague statement that the department is committed to quality. It confirms that something is being hidden and leaves the false claim in circulation - and it wastes the only asset that matters in a correction, which is credibility.
The institutional point. Confidence in a health system is built by being seen to catch your own errors early, not by never admitting one. A district that says 'our pilot found a problem and here is the fix' is more trustworthy than one that says 'the pilot succeeded' and is contradicted by its own data.