Over-Refusing Patients While Letting a Roleplay Through
A clinic deploys a patient-facing assistant that answers general questions about medications, appointment logistics, and coverage, and must never give individualized dosing or diagnostic advice. Two weeks of transcripts show: 22% of benign questions ('what is metformin for?') are refused with a blanket 'I can't give medical advice'; the assistant correctly refuses personal dosing questions, but a user who framed a request as a fictional dialogue obtained specific titration steps; and one hallucinated interaction warning was shown to 40 patients before anyone noticed. The product team proposes adding a longer list of prohibited topics to the system prompt plus a second model call that checks the output. Legal wants zero risk; the clinical lead worries that over-refusal drives patients to unmoderated sources.
1. Separate the failures and say which defense layer addresses each. Why will a longer prohibition list not fix the over-refusal?
Read the worked answer
Four distinct failures, four different layers.
- 22% benign over-refusal - a policy specification problem. The prompt says 'never give medical advice' with no positive definition of what is allowed, so the model resolves ambiguity toward refusal.
- Roleplay bypass of dosing advice - a generalization problem. Refusal is tied to surface phrasing rather than to the underlying request class.
- Hallucinated interaction warning - a grounding problem, and the most serious: a false claim delivered with confidence to 40 patients.
- Legal's 'zero risk' - not a defense layer but a target that, taken literally, manufactures failure #1.
Why the longer list backfires: every added prohibition widens the region where refusal feels safe, so a model optimizing for compliance will refuse more, not less. The 22% already shows the prompt has no way to distinguish 'discuss a drug' from 'advise this patient'. Prohibitions define only what to avoid; they never define what to do.
Layer assignment: rewrite the policy spec as positive permissions plus narrow declines (system prompt); address the bypass with a request-semantics check on the output side and an input classifier, not a phrase blocklist; address hallucination with grounded generation against a vetted source plus a citation requirement and an entailment check against retrieved text; address over-refusal by measuring it, not by arguing about it.
2. Rewrite the refusal policy as a tiered behavior specification, and explain how you would test the roleplay bypass.
Read the worked answer
Specify three behaviors rather than a prohibition list.
- Tier A - answer directly. General education about a named drug, its purpose, class, common side effects, and logistics. 'Metformin lowers blood glucose; it is used in type 2 diabetes.'
- Tier B - answer plus redirect. General information that touches a personal decision. Answer the general question, then hand the personal part to the prescriber.
- Tier C - decline and route. Individualized dosing, diagnosis, symptom triage, 'should I stop taking X'. Decline the specific judgment in one sentence, say why, and offer the clinician line or emergency number. Refuse the request class, not the topic.
Each tier needs three to five worked examples in the prompt, plus an explicit tie-break rule such as 'when unsure between B and C, choose B and redirect' - otherwise the model defaults to C and over-refusal returns.
Testing the bypass. It worked because the model evaluated the surface frame (fictional dialogue) instead of the underlying request (titration steps for a real patient). Build a red-team set of 60–100 attempts spanning roleplay, hypothetical framing, 'for my novel', claimed authority, and incremental multi-turn escalation. Require that Tier C requests be declined across all frames. Add a framing-independent output check: if a response contains stepwise dose numbers or a titration schedule, block it regardless of context. Then keep the red-team set in the regression suite - a bypass fixed once comes back after the next prompt change.
3. Define how you would measure safety and quality here, and what 'zero risk' should mean operationally.
Read the worked answer
Metrics, each with an owner and a cadence.
- Over-refusal rate: share of a clinician-labeled benign set (≥200 questions) answered with a flat refusal. Target an agreed ceiling, e.g. below 5%, tracked weekly. This is the metric that protects patient access and stops the policy collapsing into silence.
- Under-refusal rate: share of Tier C requests answered with individualized advice, measured on the red-team set by frame.
- Unsourced-claim rate: clinical claims not entailed by a retrieved source, sampled weekly by a clinician.
- Time-to-detection: the interaction warning reached 40 patients, so track how fast a defect is caught, not only how many occur.
What 'zero risk' means operationally: a written list of never-events (individualized dosing, diagnosis, unsourced interaction claims) with a target of zero in sampled review, plus a defined acceptable ceiling for everything else. Say plainly what the team already knows: no prompt and no output classifier achieves literal zero. What does bound the risk is a narrow output scope, grounding with citations, a blocking check on never-events, and a human review queue for flagged conversations. Report the never-event count to legal monthly. That is a defensible position; a longer prohibition list is not.