The Readmission Model That Looked Excellent
A 600-bed hospital deploys a vendor's machine-learning model to predict 30-day readmission. The model is trained on 18,000 historical discharges; the readmission rate is 8.4%. Validation looks excellent: AUC 0.91 and accuracy 93%. After four months live, AUC falls to 0.63. Investigators find that the training data included post-discharge fields such as 'follow-up appointment booked within 7 days' and 'medication reconciliation completed', and that a random split put multiple admissions of the same patient in both training and test sets. At the deployed threshold of 0.5, the model labels 96% of patients 'no readmission' and flags only 3% of true readmissions. Clinicians now ignore alerts, saying they are 'always wrong'. The vendor blames 'data drift' and proposes retraining on the last six months.
1. Explain why the validation results were misleading. Address data leakage, splitting, class imbalance, and the difference between accuracy and operational usefulness.
Read the worked answer
The AUC of 0.91 was inflated by leakage. 'Follow-up appointment booked within 7 days' and 'medication reconciliation completed' are post-discharge events; at the moment of prediction they are unknown, and they are also consequences of good care, so the model learned the answer rather than a usable precursor. The random split compounded this: multiple admissions of the same patient appeared in both training and test sets, letting the model memorise patient-specific patterns. The 93% accuracy is the accuracy paradox: with an 8.4% positive rate, a model that predicts 'no readmission' for everyone is already about 91.6% accurate. Accuracy therefore hides that at threshold 0.5 the model catches only 3% of true readmissions. AUC is threshold-independent but still not a deployment metric; clinicians need precision at the alert volume they can handle, recall for the cases they care about, and calibration so a score of 0.7 means roughly 70% risk. The live AUC of 0.63 suggests both leakage and temporal change; the vendor's 'drift' label is incomplete.
2. Design a rebuild and evaluation plan for the model.
Read the worked answer
First, define the prediction point precisely, for example at discharge, and exclude any feature not available then. Hold a clinical review of every feature for leakage. Second, split by patient and by time: train on earlier years, validate on a later period, and test prospectively. Third, address imbalance: report prevalence, precision-recall curves, sensitivity and specificity at candidate thresholds, and calibration (reliability diagram, Brier score). Choose the threshold from capacity and cost, not default 0.5: if the team can review 100 alerts per month, set the threshold to yield roughly that volume and measure how many true readmissions are captured. Fourth, run a silent prospective trial before alerts go live, then a stepped-wedge pilot with a comparator. Fifth, monitor input drift, performance drift, alert acceptance, and subgroup performance. Sixth, include clinicians in feature selection and workflow design so the model supports a decision rather than issuing a verdict.
3. The chief medical officer asks whether to terminate the vendor. Advise using a decision framework.
Read the worked answer
Distinguish three failures: model development (leakage), validation (patient-level split), and deployment monitoring (drift and alert fatigue). Termination is justified if the vendor refuses transparency, cannot provide a leakage-free rebuild, or hides behind 'drift'. It is premature if the vendor will open the pipeline and accept independent validation. Use a gated remediation: 30 days to deliver feature lineage, a patient-level temporal validation, calibration metrics, and a prospective silent-mode test. Require contractual rights to audit data and models, and a performance bond tied to prespecified metrics. In parallel, fix the workflow: do not show a bare probability; show the top contributing factors and the prediction time, and route alerts to a care coordinator with capacity. If the rebuild cannot beat a simple baseline such as prior admissions plus social risk, stop. The CMO's real question is not 'keep or fire' but 'what evidence would justify either'; set that evidence before the next contract decision.