The Leaderboard Winner Nobody Can Serve

Judge on the operating region, not the global rank. AUC integrates over the whole score range, including the extreme high and low tails the campaign team never acts on. Compare the candidates at the threshold actually used - precision at fixed recall, lift in the top two deciles, expected redemptions per thousand offers - and prefer the simple model if the difference there is inside the noise band. With 6.5M rows and roughly 5-fold CV, differences under about 0.005 AUC are frequently seed noise; rerun the top two candidates with three different seeds before believing the gap.

Then price the operational difference. The ensemble means twelve MOJOs to version, twelve feature-vector contracts to keep aligned, and 165 ms against a 40 ms budget - four times over, so it cannot ship as specified without a caching layer that changes the product. The single GBM at 0.8061 fits the budget and has one artefact.

A reasonable recommendation: deploy the GBM, and keep the ensemble as an offline scoring model where latency is irrelevant (e.g. the monthly campaign build). If the ensemble's uplift in the top deciles later proves material and stable, distil it - train a single GBM on the ensemble's out-of-fold scores - so you recover most of the gain in one servable artefact. Note also that the 0.0063 was measured on the same folds the metalearner was fit on; confirm it on a genuinely held-out time period before it justifies any engineering work.

Random folds mix future and past: a member's behaviour in month 22 informs a model that will be scored on members in month 3. Redemption is seasonal, campaign-driven, and affected by programme changes, so the model effectively sees the answer sheet. CV AUC is optimistically biased, and the bias is largest exactly for the models that memorise time-local patterns - which distorts the leaderboard itself, not just the reported number.

The fix in H2O-3: build an explicit contiguous time-block fold column (for example five blocks of roughly five months each) and pass it as fold_column to h2o.automl(). That stops random mixing, though it still trains on later blocks within a fold; for a genuinely forward-looking estimate, also hold out the most recent three months as a validation_frame that AutoML never sees, and report the leaderboard and the holdout numbers side by side. If the ranking flips between the two, the CV ranking is not trustworthy.

Run the final candidate comparison as rolling-origin validation: several train/validate pairs where training always ends before validation begins. It is more work than one h2o.automl() call, but for a monthly-cadence production model it is the only scheme whose estimate corresponds to the decision you are actually making.

Doubling the budget mostly re-explores the same space with a different random seed. When the top five models sit within 0.01 AUC, additional runtime buys leaderboard reshuffling, not capability - and each monthly rerun invalidates the comparison with last month's incumbent because the search space changed.

A better strategy:

1. Fix the validation scheme first (time-block folds plus a recent holdout). Tuning against a leaky metric optimises the wrong thing faster.
2. Narrow the families. Use include_algos to keep the two or three families that actually win (typically GBM and XGBoost here) and stop spending the budget on models that never place.
3. Move from broad AutoML to targeted search. Run h2o.grid with random discrete search over a small, defensible space - max_depth, min_rows, sample_rate, col_sample_rate_per_tree, learning_rate - with early stopping (stopping_rounds=5, stopping_metric='AUC'). Twenty well-chosen points usually beat hundreds of blind ones.
4. Budget by model count, not wall clock, so runs are comparable month to month, and log every run's fold scheme and seed.
5. Gate on stability. If the top candidates are within noise, choose on latency, interpretability and operational simplicity - not on the fourth decimal place of CV AUC.