Deploying a Predictive Deterioration Model on the Wards
A hospital wants to deploy a vendor early-warning model that predicts unplanned ICU transfer within 12 hours. Validation on the vendor's data showed an AUC of 0.82. Local silent-mode testing over 3 months on 8,400 admissions yielded AUC 0.74, with calibration drift: the model over-predicts risk in patients over 80 and under-predicts in post-operative patients. The proposed workflow pages the rapid response team when the score exceeds a fixed threshold, which would generate about 22 pages per day. The RRT currently responds to about 9 calls per day and reports being stretched. The model uses 41 inputs, 6 of which are frequently missing at admission.
1. How do you decide whether the model is ready for live use, and what do you do about the calibration problem?
Read the worked answer
Readiness is not an AUC question alone. The local AUC of 0.74 is lower than vendor-reported 0.82, and calibration is the more important issue for a threshold-based alert: over-prediction in the elderly will drive false pages, under-prediction in post-ops will miss the highest-acuity group. Before go-live: (1) recalibrate locally (e.g., Platt scaling or isotonic regression) and re-validate on a held-out period; (2) address the 6 frequently missing inputs - either impute with documented methods or drop and retrain, and quantify the performance impact; (3) evaluate performance at the specific operating threshold, not just AUC, reporting sensitivity, PPV, and number-needed-to-alert. Only proceed if calibration is acceptable and the alert burden is sustainable.
2. The 22 pages/day would overwhelm the RRT. What deployment strategy reduces burden without abandoning the model?
Read the worked answer
Do not page the RRT for every threshold crossing. Options: (1) raise the threshold to a level that yields a PPV the RRT can act on, accepting lower sensitivity, and pair it with a lower-severity in-EHR notification for the rest; (2) route low-tier alerts to the charge nurse or primary team rather than the RRT; (3) use a tiered escalation (score bands) so RRT only gets the highest band; (4) phase rollout to one or two units first to measure actual page volume and RRT workload. Track balancing measures: RRT call volume, response time, and whether true deteriorations are still caught. The goal is a sustainable alert-to-action ratio, not maximal sensitivity.
3. What monitoring plan keeps the model safe after go-live?
Read the worked answer
A continuous performance-monitoring plan: monthly AUC and calibration plots against confirmed ICU transfers, alert volume and override rates, PPV at the operating threshold, subgroup performance (age, surgical status, race/ethnicity, language) to detect bias, and data-drift checks on input distributions and missingness. Define pre-specified trigger thresholds for review (e.g., calibration slope outside a set band, or subgroup PPV disparity) and an owner and cadence. Include a mechanism for clinicians to report false negatives and a periodic re-validation schedule. Document model version, training data, and limitations in a model card accessible to clinical leadership.