Readmission Model with Good Discrimination but Poor Calibration
Meridian Regional Hospital has 220 beds and a 30-day readmission rate of 8.7%. A vendor-supplied readmission model reports AUC 0.81 on your 2022 validation set, but the calibration slope is 0.68: patients scored at 0.30 risk are observed to be readmitted about 41% of the time. At the 0.20 threshold the model flags 14% of discharges, while the care-management team can intervene on only 8%. A readmission costs roughly $9,800; the intervention costs about $400 per patient and is not proven to reduce risk in your population. The model was trained on 2019–2021 data, before a telehealth program changed follow-up patterns. The CMO wants to go live next month with the vendor's default 0.20 cutoff and report "high-risk" patients to the board.
1. Which operating metric and threshold would you choose, and why is AUC not sufficient here?
Read the worked answer
The model's job is to allocate 8% of discharges to a scarce intervention, so the operating metric should be precision at the capacity point and expected net benefit, not AUC. AUC measures ranking across all thresholds and is insensitive to the decision context. I would sweep thresholds and compute, for each, the number flagged, true positives, false positives, and net benefit = (readmissions prevented × $9,800) − (interventions × $400). Because the intervention is unproven, I would show a sensitivity analysis at 0%, 10%, and 25% relative risk reduction.
At 0.20 the model flags 14%, above capacity. The threshold must rise to fit 8% of discharges. But because calibration is poor, the nominal 0.20 does not mean 20% risk; predicted 0.30 corresponds to observed 0.41. I would recalibrate first on a held-out 2022 set, then choose the threshold that flags exactly 8% and report the observed readmission rate in that flagged group.
A useful sanity check: with 8.7% prevalence, random selection of 8% of discharges would capture about 8.7% of readmissions. The model must beat that substantially to justify the workflow. I would report lift over random at the same capacity, with a confidence interval.
2. What could cause the calibration slope of 0.68, and how would you diagnose and remediate it?
Read the worked answer
A calibration slope of 0.68 means predictions are too extreme: high-risk patients are overpredicted and low-risk patients underpredicted relative to observed outcomes. Possible causes include:
- Case-mix and practice shift between 2019–2021 training data and 2022, including the telehealth program.
- Base-rate mismatch: the vendor's training prevalence may differ from your 8.7%.
- Outcome-definition mismatch: all-cause versus potentially avoidable readmission, or a different attribution window.
- Missing variables that changed after 2021, such as post-discharge follow-up intensity.
Diagnosis: plot calibration by decile with confidence intervals; compare outcome definitions and inclusion criteria; check whether miscalibration is uniform or concentrated in a service line, payer, or discharge destination. If ranking is still good, recalibration on recent local data can fix the slope without rebuilding the model. If discrimination has degraded within important subgroups, recalibration alone is not enough.
Remediation: fit an isotonic or Platt calibration layer on a 2022 holdout, validate on a later period, and set a monitoring trigger such as recalibrating when the slope leaves 0.85–1.15. Clinicians should see recalibrated risk bands with intervals, not raw vendor probabilities.
3. The CMO wants a go-live next month with the default cutoff. What do you recommend and what would you tell the board?
Read the worked answer
I would not go live next month with the vendor's default cutoff and board reporting of "high-risk" patients. The default threshold exceeds care-management capacity, the probabilities are miscalibrated, and the intervention's effectiveness is unproven - three separate reasons a board dashboard could mislead.
A defensible path:
- Recalibrate on 2022 data and validate on the most recent 3–6 months.
- Set the threshold from capacity (8%) and report the observed readmission rate and lift, not vendor scores.
- Run a pragmatic pilot - randomized or stepped-wedge - with a prespecified primary outcome and a concurrent comparison group.
- Monitor calibration, discrimination, and subgroup performance monthly.
- Give the board a one-page statement of what the model does and does not establish.
If the CMO wants a board metric now, I would offer "percentage of discharges screened and reviewed" and "observed readmission rate among intervened patients versus a matched comparison," rather than "high-risk patients identified." The latter sounds like a causal claim the evidence cannot support.