Leakage in an Intraday Tree Model: Random CV Versus Walk-Forward
You are reviewing a US equities intraday alpha model built by a small systematic desk. The model is a gradient-boosted tree predicting the sign of the next 30-minute mid-price return for 40 liquid names, trained on three years of 1-minute bars. Features include order-book imbalance, signed trade flow, realized volatility, spread, and a field labelled "post-trade VWAP". With random 5-fold cross-validation the model reports AUC 0.71 and a 1.9 bps average edge per trade. The desk proposes deploying at 8 round-trips per day with an all-in cost of 1.2 bps per side. A junior quant reruns validation using walk-forward splits with a one-day embargo and gets AUC 0.53 and a net edge of -0.4 bps. Feature importance in the two runs disagrees sharply, and post-trade VWAP ranks first in the random-CV run. The desk head wants a decision by Friday: deploy, fix, or shelve.
1. Diagnose why the two validation estimates differ so much, and state what the 0.71 AUC is actually measuring. Include the cost arithmetic the desk head needs.
Read the worked answer
The gap is almost certainly leakage rather than a modelling improvement. Three mechanisms are visible:
- Forward-looking feature. Post-trade VWAP is computed after the label window closes, so at prediction time it is unavailable or contaminated. Its top rank in random CV is the classic signature of target leakage.
- Overlapping labels. A 30-minute return sampled every minute overlaps its neighbours by 29/30. Random 5-fold CV places near-duplicate rows in both train and test, so the model effectively memorises the test period.
- No temporal separation. Random CV ignores that deployment is forward in time. A one-day embargo is also too short relative to feature lookbacks such as 20-day volatility; purging must cover the full feature lookback plus the label horizon.
The 0.71 AUC therefore measures in-sample recall of contemporaneous information, not forecast skill. The honest estimate is closer to the 0.53 walk-forward result.
Cost arithmetic: at 8 round-trips per day and 1.2 bps per side, the round-trip cost is 2.4 bps. Even the inflated 1.9 bps gross edge is below cost, so the strategy loses money before any leakage correction. At the corrected edge the loss is roughly 2.8 bps per round trip, or about 22 bps per day at 8 turns.
2. Specify a corrected evaluation protocol and give a deploy, fix, or shelve recommendation with explicit conditions.
Read the worked answer
Protocol:
- Remove post-trade VWAP and any feature whose timestamp is not strictly before the prediction time; audit the feature pipeline for as-of joins.
- Use purged K-fold or combinatorial purged cross-validation with an embargo at least as long as the label horizon plus the longest feature lookback (here, weeks rather than one day).
- Apply sample weights that down-weight overlapping observations, or sample non-overlapping windows.
- Evaluate strictly net of 2.4 bps round-trip cost and report the distribution of daily P&L, not only average edge.
- Deflate the reported Sharpe for the number of model variants tried, and test feature-importance stability across folds.
Recommendation: shelve deployment now, fix and revalidate. The current evidence is negative after costs. Conditions for revisiting: a leakage-free pipeline, out-of-sample net edge comfortably above 2.4 bps (say above 3.5 bps to leave a margin), stable feature importance across purged folds, and capacity analysis at the intended trade size. Until then, no capital.
3. The desk head argues that AUC 0.53 still beats a coin flip, so a smaller position should be profitable. Respond quantitatively.
Read the worked answer
AUC 0.53 is a very thin separation, and position sizing does not repair negative expectancy. If the base rate is near 50%, an AUC of 0.53 corresponds to a probability tilt of only a few percentage points around 0.5. Translating that into expected return typically yields an edge of a few tenths of a basis point per trade, well below the 2.4 bps round-trip cost. Scaling position size multiplies both the edge and the cost, so the sign of the net expectancy is unchanged.
There is also an estimation problem. With overlapping 30-minute labels sampled every minute, the effective number of independent observations is roughly one-thirtieth of the row count. The standard error of AUC is therefore much larger than a naive calculation suggests, and 0.53 may not be statistically distinguishable from 0.50. Before discussing sizing, the desk should report a confidence interval from block-bootstrapped, purged out-of-sample predictions and the net edge distribution. If that interval includes zero and the point estimate is below cost, the correct size is zero.