The Support Bot That Refused Its Actual Users
A patient-support assistant runs every incoming message through a safety classifier before the LLM. An audit of 5,000 sampled real conversations finds: the filter blocks 7.4% of benign messages; on a 400-prompt internal red-team set it catches 62% of harmful prompts; and benign block rate for messages written in AAVE orthography or with typos is 2.6x the rate for standard-English messages. Users who hit a block churn at 3x the rate of those who do not. The team's proposed fix is to retrain the classifier on a larger red-team set to push recall above 90%.
1. Quantify the tradeoff the current filter is making. Use an explicit prevalence assumption and say why it matters.
Read the worked answer
Assume harmful messages are 1% of traffic, which is generous for a patient-support channel. Per 10,000 messages: 100 harmful, 9,900 benign.
- Harmful caught: 62% of 100 = 62. Harmful missed: 38.
- Benign blocked: 7.4% of 9,900 = 733.
- Total blocks: 795, of which 62 are correct. Precision is 7.8%. About 92 of every 100 blocks hit a legitimate user.
The prevalence assumption is the crux. At 0.1% harmful prevalence the precision falls to under 1%; at 10% it rises to about 48%. Any argument of the form 'our classifier is 62% accurate at catching harm' is missing the denominator, and the denominator is the whole decision.
This reframes the proposed fix. Raising red-team recall from 62% to 90% catches 28 more harmful messages per 10,000 while doing nothing about the 733 false blocks - and if the retrained model is more aggressive, the false block count rises. The churn multiplier makes this concrete: with 733 blocked benign users per 10,000 messages, the retention cost of the filter likely exceeds the harm it prevents at this prevalence, and that is a claim the team should test rather than assume.
2. The 2.6x disparity is raised in review. What does responsible handling of that number look like, technically and procedurally?
Read the worked answer
Technically. First disaggregate: report block rate by dialect, typo density, message length and topic, with confidence intervals, because a 2.6x ratio on a 5,000-message sample needs its uncertainty stated before it drives a redesign. Then inspect the mechanism - tokenizers and pretrained classifiers systematically penalize out-of-vocabulary orthography, so the likely cause is training data composition, not an explicit feature. The fix is benign data collection from the affected population, relabeled with the same rubric, plus evaluation stratified by the same slices. Training only on a larger red-team set will not move this number at all, because red-team data contains no benign dialect variation.
Procedurally. Do not let the disparity live in a slide. Record it as a tracked defect with an owner and a date. Since the harm is asymmetric and immediate, add an interim mitigation: tiered handling instead of a hard block, so a false positive produces a cautious answer plus a human escalation path rather than a refusal.
Finally, be careful with the framing. 'AAVE orthography' is a proxy for a user population, and the disparity is evidence of disparate impact regardless of intent - treat it as a defect in the system, not a property of the users.
3. Redesign the safety layer given that both false positives and false negatives now have measured costs.
Read the worked answer
Stop treating it as a binary gate. A single classifier threshold forces one operating point onto all traffic. Instead: a high-confidence harm band gets an immediate safe response and human escalation; a middle band gets a hedged answer with resources and an offer to connect a person; a low band passes through. This converts most false positives from a refusal into a mildly degraded but still useful interaction, which attacks the 3x churn directly without lowering the bar on real harm.
Pick the threshold from a cost model, not from a recall target. Set the cost of a false positive (churn, plus the clinical risk of a user who needed help and got a wall) against the cost of a false negative by harm category, and choose the operating point that minimizes expected cost. Report it as an explicit decision with the numbers attached.
Evaluate on production-distributed data. The red-team set measures only one axis. Maintain parallel suites: real-traffic benign samples (to monitor false positives by slice), red-team adversarial prompts, and self-harm and crisis phrasings, which deserve their own routing rather than a generic block.
Handle the crisis case explicitly. A blocked message from a user expressing self-harm risk is the worst outcome in this system. Ensure that path goes to escalation, never to a generic refusal.
Ship with monitoring: block rate by segment, disagreement rate against periodic human review, and appeal or escalation volume, reviewed on a fixed cadence with a named owner and defined rollback triggers.