The 0.994 AUC that did not survive deployment

Three independent defects stack up here, and any one of them would produce a gap this large.

1. Target leakage. days_since_last_dispute is populated only after a dispute exists, so it encodes the label. The model learned to read the answer, which is why AUC reached 0.994 - that number is a leakage detector, not a performance result. Any feature whose value depends on events after the transaction timestamp must be excluded or reconstructed strictly as-of the transaction time.

2. Temporal leakage in the split. A random 80/20 split mixes future transactions into training and past ones into test. Fraud patterns drift with fraudster behaviour, so random splits flatter the model. Use an out-of-time split - train on months 1–14, validate on 15–16, test on 17–18 - and never let a scoring timestamp precede the newest training timestamp.

3. Point-in-time correctness. account_txn_avg_24h must be computed from data available at the moment of scoring, not from a full-history aggregation. Rebuild it as an as-of join over the event stream and verify with spot checks: take ten scored transactions and recompute their features by hand from the raw events.

4. The threshold is a business parameter, not a default. With a 0.6% base rate, 0.5 is arbitrary and will produce a flood of low-precision alerts. Optimise and report PR-AUC, precision@k, and recall@k on the out-of-time test set.

Then add a monitoring baseline: the distribution of scores and the realised precision of the top 400 daily alerts, measured against confirmed frauds once labels arrive.

The two runs differ in three places, and the third is the dangerous one.

• scale_pos_weight changes the loss function, hence the fitted model. It is a hyperparameter and belongs in the experiment record, not in someone's notebook cell.
• The random seed changes the split and the stochastic training path. With 0.6% positives, seed variance alone can move AUC by a point or more, which tells you the metric is fragile and should be reported with an interval from multiple seeds, not as a single number.
• A rebuilt feature table is the one that must never be silently possible. If the second scientist read a newer partition of the source, they trained on different data. Two AUCs from different datasets are not two estimates of the same quantity; the comparison is meaningless.

Controls

1. Log every run to MLflow with git SHA, runtime image digest, all hyperparameters including scale_pos_weight, the seed, and the exact data identity (table, partition range, or snapshot ID).
2. Freeze the evaluation set as a versioned table and log its hash with each run, so scores are always computed on identical rows.
3. Make the feature table a scheduled, versioned artifact. Scientists consume a named version; they do not each rebuild it.
4. Add a small automated re-run in CI that reproduces a reference run and fails if the metric deviates beyond a tolerance. That catches unintended pipeline changes before they become two irreconcilable numbers.
5. Agree one primary metric (PR-AUC on the out-of-time set) and one secondary business metric (precision at the review capacity) before comparing anything.

The threshold is a capacity-allocation decision, not a statistical one, so read it off the precision-recall curve at the operating point you can actually staff.

Procedure

1. Score the out-of-time test set and sort by predicted probability.
2. Take the top 400 per day - that is the review capacity - and compute precision and recall at that cut. Precision is the fraction of reviewed alerts that are genuine fraud; recall is the fraction of all frauds caught. Report both, plus the absolute counts.
3. Derive the probability threshold that yields roughly 400 alerts on a typical day and check that it is stable across days; if it swings wildly, the model is not calibrated enough to use a fixed cut and you should review by rank instead.
4. Convert to money: cost of a missed fraud versus cost of a review, then check whether a higher or lower cut improves expected value. Capacity is a constraint, not necessarily the optimum.

Illustration. Suppose the portfolio processes 300,000 transactions a day at 0.6%, so about 1,800 frauds. At 400 reviews and 55% precision, reviewers confirm 220 frauds - roughly 12% recall. At 25% precision they confirm 100. That is the trade the business is actually making, and it is invisible if you only report AUC.

Report alongside

• The base rate and the capacity assumption, so the numbers are interpretable.
• Calibration: whether a score of 0.7 really means 70%.
• Precision and recall with confidence intervals, not point estimates.
• The expected daily alert volume at the chosen cut, including how it varies with seasonality.
• The retraining and review cadence, since a threshold chosen today will drift.