Triage Chatbot Across Two Jurisdictions: Classification, Frameworks and Launch Pressure
Helio Health plans to launch TriageBot in Ireland and Ohio on 1 October 2027. The chatbot asks callers about symptoms, suggests an urgency band (emergency, same-day, routine), and flags self-harm language to a nurse queue. It does not diagnose or recommend treatment. Helio’s medical director says it is “administrative.” The US partner wants marketing that says “clinically validated.” The EU launch team has an ISO/IEC 42001 gap assessment, a NIST AI RMF profile, and a draft Article 50 transparency notice. The board asks whether the system is high-risk under the EU AI Act, whether the Ohio launch changes the analysis, and whether the existing assurance artefacts are enough. No conformity assessment has been done; the vendor is a US start-up and will not disclose its evaluation data. Launch budget is fixed; delaying by three months would cost €400k.
1. Analyse whether TriageBot is high-risk under the EU AI Act. State what facts are decisive and what remains uncertain.
Read the worked answer
Start with intended purpose, not the medical director’s label. Under the EU AI Act, a system is high-risk if it falls within Annex I (product safety, including medical devices) or Annex III. TriageBot asks about symptoms and assigns urgency, and flags self-harm language to a nurse queue. If its intended purpose is to inform clinical prioritisation, it may be a medical device under Regulation (EU) 2017/745, which would make it Annex I high-risk; the fact that it does not diagnose is not decisive. If it is purely administrative-for example, a callback queue with no clinical output-it may fall outside Annex I and outside Annex III, but that conclusion needs a documented intended-purpose analysis. Annex III includes access to healthcare services, but not every healthcare-adjacent tool is caught. Article 50 transparency applies to the chatbot interaction: people must be told they are interacting with an AI system. The Ohio launch does not change the EU classification if outputs are used in the EU, but US marketing claims such as clinically validated can create separate FDA and FTC exposure. The board should be told the classification is conditional on intended purpose and on whether the self-harm flag is a clinical function. If high-risk, launch on 1 October 2027 requires conformity assessment, CE marking, registration and post-market monitoring; the current file has none.
2. Evaluate the claim that the ISO/IEC 42001 gap assessment plus a NIST AI RMF profile make the system “AI Act compliant.” What gaps remain?
Read the worked answer
ISO/IEC 42001 certifies an AI management system-governance, policy, risk processes, lifecycle controls-not the conformity of a specific product. NIST AI RMF is a voluntary risk-management framework organised around govern, map, measure and manage; it helps structure evidence but does not create legal compliance. Neither artefact proves that TriageBot meets the AI Act’s essential requirements for high-risk systems, nor that the provider has carried out a conformity assessment. The gap assessment should be treated as a starting point. Remaining gaps include: a formal intended-purpose statement; high-risk classification and, if applicable, medical device classification; technical documentation; data governance and testing for bias, accuracy and robustness, including self-harm flag performance across languages and accents; human-oversight design with authority to override; logging and post-market monitoring; incident reporting; and a fundamental rights impact assessment if Helio is a qualifying deployer. The vendor’s refusal to disclose evaluation data is a material gap: without it Helio cannot independently verify claims or meet its deployer duties. The existing artefacts are useful evidence of process maturity, but they do not equal compliance. The board should see a gap register with owners and dates, not a certificate.
3. Recommend a launch decision with conditions, and explain how the decision should be documented.
Read the worked answer
Recommend a conditional pause of the direct-to-patient launch and a limited nurse-facing pilot only if the intended purpose is redefined and the classification analysis supports it. The decision logic: if TriageBot is high-risk under Annex I or III, it cannot be placed on the market or put into service on 1 October 2027 without the required conformity assessment, CE marking, registration and instructions for use; a €400k delay is not a lawful substitute. If it is not high-risk, Helio still needs Article 50 transparency, a GDPR lawful basis, a DPIA, clinical safety review, and US FDA/FTC review of the clinically validated claim. Conditions for any launch: remove unverified marketing claims; publish a clear AI disclosure; define the nurse override and escalation pathway; test the self-harm flag for false negatives; retain logs; contract for incident notification and audit cooperation; and set a kill switch. The decision record should state the facts relied on, the classification conclusion, the residual risk accepted by the accountable executive, and the trigger for re-review. Because the vendor will not disclose evaluation data, Helio should either obtain independent test results or restrict the tool to a non-clinical administrative queue. The board should be asked to approve the residual risk explicitly, not just the launch date.