Hybrid Adverse-Event Extraction With Rare Drugs
MedSignal extracts adverse-event entities from 8,000 pharmacovigilance reports. Fourteen entity types include drug, dose, frequency, route, and reaction. A gazetteer of 12,000 drug names has precision 0.94 but recall 0.51; a CRF trained on 6,500 reports has overall entity F1 0.78 but rare-drug F1 0.42. Two annotators disagree on 18% of dose spans because '500 mg twice daily' is sometimes labeled as one dose span and sometimes split into dose and frequency. Overlapping entities occur in 30% of reports, for example 'IV' as route inside a drug name. The team has budget for 1,200 new annotations and one engineering sprint. A vendor offers a biomedical transformer with F1 0.86 on a public corpus but no local validation and 12× inference cost.
1. Design a hybrid gazetteer-plus-CRF architecture that improves rare-drug recall without destroying precision. Explain why simple union is not enough.
Read the worked answer
The two systems fail in complementary ways: the gazetteer has precision 0.94 but recall 0.51, while the CRF has overall F1 0.78 but rare-drug F1 0.42. Simple union is not enough because it imports the gazetteer's false positives and does not resolve overlapping spans. Use a high-precision dictionary as a candidate generator and the CRF as a contextual filter. Build a trie or Aho-Corasick matcher with normalization for salts, brand/generic aliases, and known misspellings; add fuzzy matching only with a high threshold and a context check. Feed dictionary-match features into the CRF: match length, source, distance to a dose token, and whether the match is inside a section header. Add character n-grams, word shape, and dependency/path features. For overlaps, use a transition-constrained CRF or a small span classifier to resolve route-inside-drug cases. Add consistency rules: a dose must have a unit, and a frequency must be in a known set. The biomedical transformer may help rare drugs, but at 12× cost and without local validation it should be a teacher for distillation or applied only to low-confidence spans after the hybrid.
2. Allocate the 1,200 new annotations across guideline adjudication, rare-drug coverage, and testing. Justify the allocation.
Read the worked answer
Spend annotations where error and ambiguity are highest. First, allocate 300 reports for double annotation of dose and frequency to measure and fix the 18% disagreement; write a decision rule such as '500 mg twice daily' = dose span '500 mg' plus frequency span 'twice daily', and adjudicate disagreements. Second, allocate 500 reports enriched for rare drugs and overlapping entities, selected by uncertainty sampling from the CRF (lowest marginal probability) plus dictionary-missed candidates. Third, allocate 400 reports to a clean held-out test set stratified by entity type and rarity; never train on it. Use active learning in two rounds with retraining between rounds. Measure inter-annotator agreement by entity and span exact match, not only token F1. Do not spend all annotations on random sampling: rare-drug F1 of 0.42 will barely move. Keep a frozen guideline document and a labeling interface that enforces span boundaries; ambiguous cases go to a weekly adjudication queue. This allocation improves the decision boundary, the rarest cases, and the credibility of the final estimate.
3. Define the evaluation protocol for choosing among gazetteer, CRF, hybrid, and transformer. Include the metric that matters most for pharmacovigilance.
Read the worked answer
Evaluate with entity-level exact-match precision, recall, and F1, plus relaxed span F1 and per-entity breakdowns. Report rare-drug F1 separately and include a confusion matrix for dose versus frequency. Use the held-out test set and bootstrap confidence intervals. Because 30% of reports contain overlaps, measure overlap-resolution accuracy, not only flat F1. Compare four systems on the same test set: gazetteer, CRF, hybrid, and transformer if affordable, and include a cost-weighted metric such as F1 per 1,000 reports at a fixed latency. For pharmacovigilance, recall on drugs and reactions matters more than overall F1 because a missed serious reaction is the highest-cost error. Set a recall floor, for example 0.90 for serious reactions, and measure precision at that floor. Do a slice analysis by report source and template because templated reports inflate scores. Finally, run a prospective shadow test on 200 incoming reports with pharmacist review to catch distribution shift before changing the production extractor.