Sepsis Alert: Calibration, Capacity, and a 61% Override Rate
A 180-bed community hospital licenses a sepsis prediction model. Vendor validation reports AUC 0.86 and sensitivity 0.81 at a 0.5 cutoff, measured on a tertiary-academic, ICU-heavy cohort. Local validation on 9,200 inpatient encounters yields AUC 0.83, but the calibration plot shows a predicted risk of 0.20 corresponding to an observed sepsis rate of 0.07. At the vendor's recommended threshold the alert fires about 45 times a day. Nurses override 61% of alerts. Two near-miss sepsis cases last quarter occurred in patients who had been alerted and then overridden. The CMO wants the alert "turned down" because of alarm fatigue; the sepsis coordinator wants it "turned up." There is no spare nursing capacity for additional screening.
1. Explain the calibration gap and what it implies for the vendor's fixed 0.5 threshold. What would you do before changing any threshold?
Read the worked answer
The vendor validated on an ICU-heavy tertiary cohort; this is a community hospital with a different case mix and a lower sepsis prevalence. AUC is invariant to prevalence and to any monotone rescaling of scores, which is exactly why the two AUCs (0.86, 0.83) look reassuring while calibration has moved: a predicted 0.20 against an observed 0.07 is roughly a three-fold over-prediction.
Every downstream action uses absolute risk, not rank, so the fixed 0.5 cutoff is now operating far lower on the true-risk scale than the vendor intended. The consequence is visible in the alert volume: many alerts, most of them false, hence the override rate.
Before touching the threshold, recalibrate on local data - isotonic or Platt scaling fitted on a held-out local set - then re-derive the operating characteristics at each candidate cutoff. One caution: check whether the lower local prevalence is real or reflects weaker sepsis ascertainment on med-surg floors. Recalibrating against under-ascertained labels would teach the model that genuine sepsis is normal, which is a far worse failure than over-alerting.
2. How would you actually choose an operating threshold? Name the inputs you need and the analysis you would run.
Read the worked answer
Choose from net benefit, not from a round number.
Inputs. The recalibrated local risk distribution; the number of patients above each candidate threshold per day; the realistic review capacity; and the relative harms. A false negative means delayed antibiotics - mortality climbs steeply with each hour of delay in septic shock. A false positive costs a few minutes of nurse assessment, but at high volume it also produces desensitization that degrades response to every alert on the unit, which is a real harm and the one currently being felt.
Analysis. Tabulate, over a grid of thresholds: alerts per day, PPV, sensitivity, and number-needed-to-alert per case detected. Overlay the capacity line - any threshold that generates more alerts than the unit can review is effectively infinite, because unreviewed alerts are not alerts. Run decision-curve analysis across the plausible range of "worth it" threshold probabilities to show net benefit rather than arguing about sensitivity alone.
Likely answer. Well above 0.5 for the floor population, and a separate, higher threshold for ICU patients where the prior is different. Publish the chosen trade-off jointly with the CMO and the sepsis coordinator so it is a shared clinical decision rather than a modeler's.
3. What do you make of the 61% override rate and the two near-miss cases? How would you investigate before adjusting the model?
Read the worked answer
A 61% override rate is a signal about the alert at least as much as about the nurses, and treating it as non-compliance would discard the most useful data in the building.
Three hypotheses. (1) The alert fires on patients the clinician has already recognized and treated - the override is correct, and the alert is simply late. (2) It fires on low-specificity triggers, such as a single abnormal vital sign. (3) It duplicates other alarms on the unit, so it gets dismissed as noise.
Investigation. Pair each alert with contemporaneous chart actions to classify overrides as appropriate (clinician already acted) versus missed. Examine which trigger features dominate overridden alerts. Check subgroup sensitivity - elderly, dialysis, and immunosuppressed patients often present atypically, and feature-heavy models systematically under-rank them. The two near-misses should be reviewed individually: did the model rank them low, or rank them high and get dismissed?
Likely remedy. Alert content and workflow placement - actionable, specific, deduplicated, delivered to the right role - will move the override rate more than any threshold change. Threshold adjustment without this work just shifts which alerts get ignored.