0.94 in the Notebook, 0.68 in the Ward

1. Target leakage from post-outcome features (largest contributor). 'Follow-up appointments attended' and 'total billing amount' are only fully known after the prediction window has begun or closed. In the historical extract they are recorded and correlate strongly with readmission. At scoring time they are unavailable or defaulted, so the model's main signal simply disappears in the pilot. This is a feature-availability failure, not a statistical one, and it typically explains most of a gap of this size.

2. Preprocessing fitted before the split (moderate). Mean imputation and standardisation computed over the whole extract put test-set information into the training pipeline. The optimistic bias is usually small (order of one point or less), but it is real and it is trivially avoidable.

3. SMOTE before the split (moderate, class-specific). Synthetic minority points are interpolated between rows that end up on both sides of the split, so test rows have near-duplicates in the training set. This inflates minority-class precision/recall more than AUC; it makes the model look like it generalises to patients it has effectively already seen.

4. Random split ignoring temporal order (moderate). Readmission risk drifts seasonally and coding practice changes. A random split lets the model borrow period-specific patterns available in both halves. A temporal split is the only one that mimics deployment.

5. Genuine distribution shift (real but second-order). The pilot is last month's population through a live feature pipeline. Some of the 0.26 gap is true drift.

On the 'small-sample noise' claim: with a few thousand pilot discharges, the 95% CI on AUC is on the order of +/-0.03. A gap of 0.26 is not noise. The data scientist has identified a real phenomenon and assigned it the wrong magnitude.

Defect	Test activity	Stage	Artefact
Post-outcome features	Feature availability / time-travel test: for each feature, reconstruct its value for a historical scoring timestamp using only events with event_time <= t0	Data preparation, before training	A pass/fail flag per feature plus a documented 'available at inference' column; any feature that cannot be reconstructed is blocked
Preprocessing leakage	Pipeline unit test: assert fit() is called only on training folds and transform() on test folds. Concrete trick: place an extreme synthetic outlier in the test fold and check that predictions on other test rows do not move	Development testing of pipeline code, in CI	An automated test in the build, failing on regression
SMOTE before split	Split-integrity test: exact-hash and nearest-neighbour duplicate detection across train and test	Data preparation	Duplicate/similarity report with a threshold assertion; zero cross-split duplicates required
Random split	Temporal validation: hold out the most recent N months, report random-split and temporal-split metrics side by side	Model evaluation	A two-column metrics report; a large gap is itself a finding, not a footnote
Pilot vs notebook gap	Feature-skew / shadow-replay test: replay real traffic through both the offline evaluation pipeline and the production feature pipeline and diff the feature vectors	Pre-deployment / ML system testing	A skew report per feature (distribution distance, null-rate delta) - this is the test that directly explains 0.68 versus 0.94

Point of control: the first two are cheap and belong in CI, run on every pipeline change. The last two require a realistic deployment environment and belong to system-level testing, repeated at each retrain.

Proposed criterion (illustrative numbers, to be agreed with clinicians):

On a temporally held-out period of at least three months, using only features computable at the scoring timestamp, and executed through the production feature pipeline:

• AUC >= 0.80, reported with a 95% confidence interval;
• recall >= 0.70 at a precision floor of 0.30 (the case-management capacity constraint);
• no feature whose event time is after t0;
• metrics stable within +/-0.03 of a shadow-mode replay on live traffic;
• results reported per site and per key subgroup, not pooled only.

Why each clause exists

• Temporally held out - forbids the random split that let the model borrow period-specific patterns.
• Only features computable at t0 - kills the target leakage at the requirement level rather than by review.
• Production feature pipeline - the 0.68 came from the pipeline, not the model; the evaluation must exercise the same code path.
• Confidence interval - forces the 'small sample' argument to be made quantitatively.
• Precision floor plus recall - ties the threshold to operational capacity; a bare AUC hides the operating point entirely.
• Shadow-replay stability - turns the notebook-versus-ward gap into a monitored acceptance condition.

Also required: the model card must state the label definition, the labelling window, and the population the model was trained on, so the next reviewer can see the difference between the offline and deployment distributions without re-deriving it.