The Stalled Triage Model After a Vendor Dataset Refresh
A hospital's radiology AI team deployed a chest X-ray triage model nine months ago. Internal validation showed AUROC 0.93 and sensitivity 0.91 at the chosen threshold. Last month the vendor supplying the pretraining corpus pushed a 'quality refresh' that silently re-labeled 4% of images and removed 11,000 pediatric studies. The team retrained on the refreshed corpus plus their own 6,200 local studies, and the new model reports AUROC 0.94 on the same held-out split. However, the local quality dashboard shows that among 1,450 real-world cases in the first two weeks, sensitivity on patients under 18 fell from 0.89 to 0.71, while overall sensitivity rose slightly to 0.92. The clinical lead wants to ship the new model because the validation numbers improved. The ML engineer notes the held-out split was drawn from the same refreshed corpus.
1. Diagnose why the validation result and the production dashboard disagree, and state what the team should do before any further deployment decision.
Read the worked answer
The two numbers measure different populations. The held-out split was drawn from the refreshed corpus, so it inherited the vendor's re-labeling and the removal of pediatric studies. Removing 11,000 pediatric studies means the held-out set now contains few or no pediatric cases, so the reported AUROC of 0.94 is computed on a distribution that no longer matches the hospital's patient mix. A 0.94 on that split is not evidence of improvement; it is evidence that the test set drifted with the training set.
The dashboard, by contrast, samples actual patients, including the under-18 group. Sensitivity there dropped from 0.89 to 0.71, a 18-point fall, which is consistent with the model never having learned or retained pediatric features after those studies were removed.
Before any deployment decision the team should:
- Freeze the current model in place and treat the new one as unvalidated.
- Rebuild a held-out set from local, clinically adjudicated cases, stratified by age band, and explicitly include pediatric cases.
- Re-evaluate both models on that local set at the operating threshold, not just AUROC.
- Audit the 4% re-labeling to determine whether it changed the ground truth in ways that matter clinically.
- Only then decide whether to retrain, and if so, whether to restore pediatric data or exclude pediatric use from the intended scope.
2. The clinical lead argues that overall sensitivity rose to 0.92, so the model is safer. Construct the counterargument using the numbers given.
Read the worked answer
Overall sensitivity is a weighted average, and the weights are the case mix. If the under-18 group is a modest share of the 1,450 cases, a large drop in that subgroup can be masked by small gains elsewhere. Suppose roughly 15% of cases are pediatric, about 218 patients: sensitivity there fell by 0.18, costing about 39 missed findings, while the adult group's sensitivity must have risen enough to pull the total up to 0.92. The aggregate looks better precisely because the harm is concentrated in a minority subgroup.
This is the classic failure mode of pooled metrics: they are insensitive to subgroup degradation when subgroups are unequal in size. A model that misses 29% of pediatric findings instead of 11% is not safer for a child in the emergency department, regardless of the headline number.
The correct framing is that overall sensitivity and subgroup sensitivity answer different questions. The clinical lead's claim is only defensible if pediatric use is explicitly out of scope and enforced in the workflow. If the model is applied to all ages, the relevant safety metric is the worst-performing clinically meaningful subgroup, not the average.