Choosing Between a Complex Model and a Simple One Under Distribution Shift
A hospital readmission model is being chosen between a regularized logistic regression (validation AUC 0.79) and a deep neural network (validation AUC 0.84) trained on three years of data from 12 hospitals. A new 13th hospital with a different patient mix and coding practices will go live next quarter. The team has 8 weeks and one ML engineer. The DNN required extensive hyperparameter tuning and its performance was sensitive to the random seed (AUC varied 0.80-0.85 across seeds). The logistic model's AUC varied 0.78-0.80. Regulatory review requires an explainable model and a documented validation report.
1. Weigh the evidence and recommend a model, justifying with the specific numbers and constraints.
Read the worked answer
The DNN's headline advantage is +0.05 AUC on validation, but its seed variance spans 0.80-0.85, so the 'true' expected performance is uncertain and overlaps the logistic model's upper range. The logistic model's variance is 0.78-0.80 - tighter and more predictable. With 8 weeks, one engineer, and a new hospital with a different distribution, robustness and explainability dominate a marginal in-distribution gain.
Recommendation: ship the logistic regression as the primary model, and treat the DNN as a research track. The +0.05 gain is not established beyond seed noise, and the new hospital is out-of-distribution, where a heavily tuned model is more likely to degrade. The explainability and documentation requirements are also cheaper to satisfy with a linear model. This is a risk-adjusted choice, not a claim that the DNN is worse in general.
2. How should the team validate that the chosen model will transfer to the 13th hospital?
Read the worked answer
Run leave-one-hospital-out cross-validation: train on 11 hospitals, validate on the held-out 12th, rotate. Report the spread of AUC across held-out hospitals, not just the mean. If performance on some hospitals drops sharply, that reveals site-specific dependence the pooled CV hid.
Additionally, check covariate shift: compare feature distributions (e.g., coding frequencies, age mix) between the 12 hospitals and any early data from the 13th. If shift is large, consider recalibration of the intercept on a small 13th-hospital sample, and monitor post-deployment performance with a pre-registered alert threshold. Document that the pooled estimate is optimistic relative to a new site and set expectations accordingly.
3. The DNN's seed variance is partly due to tuning on the validation set. Explain the methodological problem and how it biases the comparison.
Read the worked answer
If hyperparameters and early stopping were selected using the same validation set that produced 0.84, that number is optimistically biased - the validation set has been used for model selection, so it is no longer a clean estimate of generalization. The DNN had many more tuning decisions, so it accumulates more selection bias than the lightly tuned logistic model; part of the 0.05 gap may be this bias rather than real skill.
The correct comparison uses a nested scheme: inner loop for tuning, outer loop for evaluation, or a separate untouched test set. Reporting the seed range partially exposes the instability but does not remove selection bias. The team should re-evaluate both models on a held-out test set never used for tuning before drawing conclusions.