AI Transformation Leader study materials
AI Transformation Leader study resources

The Ethics Board That Could Not Say No

AI Transformation Leader · Put your knowledge to work in context.

All case studies

The Ethics Board That Could Not Say No

A health-services company formed an AI ethics review committee eighteen months ago. It has reviewed 34 proposals and approved 34, sometimes with 'conditions' that no one verifies. Membership is voluntary and drawn from legal, data science, and HR; there is no clinical representation and no community or patient voice, despite the company operating patient-facing triage tools. Meetings are monthly and proposals are reviewed in 20-minute slots. A recent proposal - an AI tool that prioritizes outreach to patients with chronic conditions - was approved in one meeting. An internal audit later found the tool's training data underrepresented rural patients and that no one had assessed whether deprioritized patients would receive worse care. The committee chair says the process 'works' because nothing has gone wrong publicly. The CEO is proud of the committee and cites it in investor materials.

1. Evaluate the committee's effectiveness. What does the 34-for-34 approval rate tell you, and what does it not tell you?

Read the worked answer

What 34-for-34 tells you: a review body that has never declined, deferred, or materially reshaped a proposal is not performing review - it is performing ratification. Combined with 20-minute slots and unverified conditions, the process lacks the three things a review needs: time to examine, authority to say no, and a feedback loop that checks whether conditions were met. The chronic-care tool's approval in a single meeting, with a data-representativeness gap and an unexamined harm to deprioritized patients, is the predictable output.

What it does not tell you: that the committee is useless, or that the company is acting in bad faith. A 100% approval rate can also reflect that sponsors self-censor before submitting, or that only low-risk proposals reach the committee. Those are worth checking - but the chronic-care case suggests the committee is not catching material issues, so self-censorship is not the explanation here.

The chair's 'nothing has gone wrong publicly' is the weakest argument in the room. It is a lagging indicator that measures detection, not safety. The internal audit did find a problem; it simply had not surfaced externally yet. Absence of a public incident is evidence about luck and disclosure, not about the quality of review.

The investor-material citation raises the stakes. The CEO is publicly representing a control that is not functioning as described. That is a governance and disclosure risk independent of any patient harm.

2. Redesign the committee's mandate, composition, and process. Be specific about authority, membership, and decision rules.

Read the worked answer

Mandate: shift from 'review and approve' to risk-tiered review with binding authority. The committee should own a risk classification (e.g., patient-facing, vulnerable populations, automated decisioning = high risk) and a decision rule that high-risk proposals cannot deploy without its sign-off, with an escalation path to the board when the committee and a business line disagree.

Composition: the current membership is the core defect for a patient-facing company. Add:

  • Clinical representation with authority to assess care-quality impact.
  • Patient/community voice - at minimum a patient advocate or community representative with voting status, not a courtesy attendee.
  • An independent member outside the company to reduce capture.
  • A data/statistics expert able to interrogate representativeness and subgroup effects.

Voluntary membership is acceptable only if time is protected and recognized in performance objectives; otherwise review is the first thing dropped.

Process:

  • Tiered review with proportionate time. 20 minutes is adequate for low-risk tools and inadequate for patient triage. High-risk reviews get pre-read materials, a structured harm assessment, and a multi-session path where needed.
  • A required harm-and-equity assessment covering subgroup performance, who is deprioritized, and what happens to them - the exact gap the audit found.
  • Conditions with owners, dates, and verification. A condition nobody checks is a note, not a control.
  • Decision options beyond approve/reject: approve with monitoring, time-limited approval, defer for evidence, and decline. A body with only one effective option is not deciding.
  • A published decision log (internally, at minimum) so decisions are reviewable.

Measure the committee by: proportion of proposals materially changed or deferred, conditions verified on time, and issues caught internally before audit or external discovery.

3. The CEO cites the committee in investor materials. What should you advise, and what is the near-term handling of the chronic-care tool?

Read the worked answer

Advice to the CEO: stop citing the committee as a control until it functions as one. Investor materials describing a review body that has never declined anything, with no clinical or patient representation, create exposure if a patient-harm event later emerges - the disclosure becomes evidence that the company knew it had a control and misdescribed its strength. The honest framing is 'we are strengthening our AI governance,' which is defensible and still positive. The reputational risk of overclaiming exceeds the benefit of the claim.

Near-term handling of the chronic-care tool - sequence:

  1. Pause or constrain deployment while the representativeness gap and the deprioritization harm are assessed. This is not a hypothetical: the tool changes which patients get outreach, and rural underrepresentation plus unexamined deprioritization is a plausible care-quality harm.
  2. Quantify the harm. Re-run performance by rural/urban and other subgroups; determine whether deprioritized patients experience delayed outreach and what that means clinically. Engage clinical leadership, not just data science.
  3. Remediate or retire. Options include retraining with representative data, adjusting the prioritization logic, or adding a human review layer for deprioritized patients. Decide with clinical input and document the rationale.
  4. Re-review through the redesigned process, and treat this case as the pilot for the new governance - it will surface exactly the gaps that need fixing.
  5. Report to the board, because the audit finding plus the investor citation is a board-level matter, not a committee matter.

Avoid the temptation to treat this as a documentation gap. The audit found a design gap: the tool was never assessed for who loses. That is the question the committee existed to ask and did not.