The Sepsis Score Everyone Ignores

Start with arithmetic, not modeling. Local sepsis prevalence is 2,900 / 41,000 ≈ 7.1%. If the score ranked patients perfectly and still flagged 31% of admissions, the best achievable PPV would be about 7.1 / 31 ≈ 23%. The observed PPV of 6% is actually below prevalence, which means the flagged patients are, on average, no sicker than a randomly chosen inpatient. At this operating point the alert carries essentially no information.

Then separate the two explanations:

• Operating point. Recompute PPV and sensitivity across every candidate cut point, and express the cost as alerts per nurse per 12-hour shift. Pick the cut point that fits an explicit alert-volume budget rather than the vendor default.
• Input timing. If lactate and cultures are missing at scoring time, the model imputes - check the imputation rate by hour and plot the score distribution for patients with versus without labs available at t=0. A score built mostly on imputed values in the first hours is a deployment problem, not a model problem.
• Workflow. Pull alert acknowledgment logs. Alerts dismissed in under 5 seconds without opening the chart indicate the threshold, not the science, is failing.

Only if local discrimination remains poor at the best cut point (compute local AUC with confidence bounds) does retraining become the right answer.

The sample size looks adequate - 2,900 events against roughly 40 candidate predictors is about 70 events per variable - so overfitting from small n is not the main risk. The real risks are label contamination and temporal validity:

1. The alert changed the labels. Every admission in this window was exposed to an alert that drove lactate draws, cultures, and escalation of care. Training on that data teaches the model to predict who the previous alert caught, not who is septic. This is indication bias, and it inflates apparent performance.
2. The outcome definition must match. CDC Adult Sepsis Event criteria (suspected infection + organ dysfunction) are not the same label the vendor used. Decide the target first, then build the label pipeline, then evaluate.
3. Only 14 months, all post-deployment. There is no clean pre-alert period for a valid holdout, and the window likely spans order-set and formulary changes. Split temporally (train on months 1–10, test on 11–14), never randomly.

aRecommendation: recalibrate the threshold now, fix the input-timing issue, and run a silent (shadow) mode for 8–10 weeks that scores every admission without displaying an alert. That produces an uncontaminated, timely evaluation set and tells you whether retraining is even necessary. Commit to a retrain decision only after shadow-mode results are in.

Agree on the metrics before the intervention, and write them into the minutes.

Primary process measures: PPV at the chosen threshold; alert volume per 100 admissions and per nurse-shift; median minutes from score generation to first antibiotic and to first fluid bolus; proportion of alerts acknowledged within 15 minutes.

Outcome measures (secondary, with the caveat that the hospital is unlikely to be powered for mortality): sepsis mortality (observed vs expected), ICU transfer within 24 hours, total antibiotic days of therapy.

Balancing measures (the ones committees forget): total alert burden across all alert types, nurse-reported override rates, C. difficile incidence, and time-to-antibiotic in the non-sepsis chest-pain and post-op populations where aggressive fluid/antibiotic pathways can cause harm.

Design: a stepped or interrupted time series with at least 6 pre- and 6 post-period months, because a simple before/after will be confounded by seasonality (winter respiratory season) and by any concurrent initiative. Pre-specify that a threshold change alone will be judged on PPV and alert volume, not on mortality, since a threshold change cannot plausibly move mortality within one quarter. Report confidence intervals, not point estimates, on every rate.