Readmission Model With Equal AUC and Unequal Recall
A health system deploys a vendor model scoring 40,000 patients per month for 30-day readmission risk and routes the top 200 to a transitional-care program. Validation shows overall AUC 0.81, with similar AUC within each demographic subgroup. But at the deployed cutoff, sensitivity is 0.68 for commercially insured English-speaking patients and 0.42 for Medicaid patients with limited English proficiency; PPV is 0.31 overall. The top features are prior admissions, prior no-shows, insurance type, and distance to clinic. Care management reports that LEP patients who do enroll do better than any other group once they get in. The vendor argues the model is fair because AUC is equal across groups and because it uses only clinical and administrative data. The CMIO wants a decision before the next contracting cycle.
1. Is the vendor correct that equal AUC across groups means the model is fair?
Read the worked answer
No. AUC is threshold-free - it measures ranking, not the error rates at the cutoff actually deployed. Two groups can share an AUC and still have different score distributions, so one threshold yields sensitivity 0.68 versus 0.42. That is a failed equalized-odds test, and the deployed decision, not the AUC, determines who receives care.
The 'clinical and administrative data only' defense misses the mechanism. Prior no-shows and insurance type encode access barriers, not need. If LEP patients historically had fewer recorded visits because of transport and language access, the model reads unmet need as low risk - the feature is a proxy for the very barrier the program exists to overcome. The label has the same defect: readmission is observed only for people who reach a hospital, so patients who cannot navigate access can look low-risk. Equal subgroup AUC computed on small subgroup samples may also be noise. Fairness has to be assessed on the outcome variable that is deployed.
2. Given a hard capacity of 200 enrollments per month and evidence that enrolled LEP patients benefit more, how should the threshold be set?
Read the worked answer
Begin from the objective: 200 slots, maximize readmissions prevented, with the team's own evidence that enrolled LEP patients benefit more once admitted to the program. That justifies benefit-weighted ranking rather than matching selection rates. Estimate the intervention effect per subgroup, score each patient as risk × expected effect, rank on that, and evaluate group-specific thresholds against the current global one.
Then make the policy choice explicitly between two defensible options: equalize recall across groups, which delivers equal access at the cutoff and forgoes some total readmissions prevented; or maximize total benefit, which accepts unequal recall. Publish the choice and its numbers, because that is a policy decision, not a statistical one, and it belongs to clinical leadership rather than the vendor. Separate the two possible failures first: if ranking within each group is sound and only the threshold is wrong, group-specific cutoffs are the cheap fix and retraining is unnecessary. Dropping insurance type is worth testing, but if the bias lives in the world, distance and visit history will reproduce it.
3. What does a credible post-deployment monitoring plan look like for this model?
Read the worked answer
Watch calibration and subgroups, not overall AUC. Quarterly: reliability curves and calibration-in-the-large by subgroup, sensitivity at the deployed cutoff by subgroup, PPV against the 200-slot capacity, and population stability on the top features. Track how the LEP flag is captured - a registration workflow change can silently reclassify patients and move measured disparity without the model changing at all.
Guard the label against intervention contamination. Enrolled patients receive care that lowers readmission, so the model learns that high prior utilization predicts lower future risk and degrades precisely where it worked. Keep a holdout that never receives the program, or model the counterfactual, to keep labels clean. Add an override log and review a monthly sample of overridden cases; if clinicians systematically override for one group, that is a finding about the model, not about the clinicians. Finally, re-run the fairness audit whenever features, thresholds, or the enrollment workflow change.