AUC 0.78 in Cross-Validation, 0.54 Out of Sample
An equity team builds a classifier predicting whether a stock will beat its sector ETF over the next 20 trading days. The dataset is 2010–2024 daily, about 1.4 million stock-days, with 1,150 features from prices, fundamentals and two alt-data vendors. The pipeline median-imputes and z-scores every feature using statistics computed over the entire sample, then runs shuffled 5-fold cross-validation: AUC 0.78, accuracy 71%, top-decile spread 4.2% per 20 days. The head of research reruns the same model as a purged walk-forward with a 20-day embargo: AUC 0.54, rank IC 0.012, top-decile spread 0.3% per 20 days. The team argues that 0.54 is 'still overwhelmingly significant with 1.4m observations' and requests a live allocation. Two features are the stock's index weight read from the current Russell 2000 file, and trailing-twelve-month EPS from a fundamentals snapshot downloaded in 2025.
1. Identify at least three distinct leakage channels in the original pipeline and state the direction of the bias each creates.
Read the worked answer
Full-sample preprocessing. Imputation medians and z-score means/standard deviations computed over 2010–2024 encode information from the future into every training row. The bias is usually small in magnitude but is unambiguously optimistic, and it is invisible in a shuffled CV because the leak is in both folds.
Shuffled folds with 20-day overlapping labels. A stock-day on 3 March and one on 10 March share most of their 20-day forward window. Shuffling puts near-duplicates in train and test, so the model is effectively scored on rows it has nearly seen. This is usually the largest contributor to an AUC gap of this size.
Point-in-time violation on index weight. Reading the current Russell 2000 file and applying it backwards means the model knows, in 2011, which companies will still be index members in 2025. Survivorship plus membership look-ahead both inflate results, and the effect is concentrated in the small caps that drive the top decile.
Restated fundamentals. A 2025 snapshot contains revised EPS. Revisions correlate with subsequent realised performance, so the feature is partly a function of the outcome. Optimistic bias.
All four channels push in the same direction, which is why the honest interpretation of the 0.78 is not 'great model, bad validation' but 'the 0.78 measures the leakage'.
2. The team says AUC 0.54 is significant with 1.4m observations and wants to allocate. Evaluate that argument with the numbers given.
Read the worked answer
Statistical significance is not the question. With roughly 1.4m overlapping observations, standard errors are tiny and almost any non-zero effect is 'significant'. The overlapping 20-day labels also mean the effective sample size is far smaller than 1.4m - closer to the number of independent 20-day windows - so even the significance claim needs a block/Newey–West correction before it is taken at face value.
Economic significance is the question. A rank IC of 0.012 and a top-decile spread of 0.3% over 20 days are the numbers that matter:
- Gross edge per 20-day holding in the top decile: 0.3%.
- Round-trip cost in liquid large caps: roughly 10–20bp; in the small caps this model favours, plausibly 30–60bp including impact.
- Net edge: somewhere between −30bp and +20bp per position, before any turnover cost from signal churn.
- A 0.3% spread also has to survive the standard error of a decile spread with cross-sectional correlation; it will not be reliably distinguishable from zero.
Verdict. The out-of-sample result is a falsification of the 0.78, not a mild downgrade. Ship nothing. The correct next step is to establish a pre-committed hurdle (e.g. net-of-cost IC t-statistic above 3 on non-overlapping windows) and only then discuss sizing. Note also that the walk-forward is still contaminated by the point-in-time violations in the feature set, so 0.54 is an upper bound on the true signal.
3. What single diagnostic would you run first to decide whether the residual 0.54 reflects a weak-but-real signal or leakage that survived the purge?
Read the worked answer
Run the pipeline on label-permuted data, with the same block structure. Shuffle the forward-return labels within each date (or across dates in blocks of 20 days), rerun the full walk-forward, and repeat 50–100 times. If the resulting distribution of AUC is centred on 0.50, the validation machinery is clean and 0.54 is a genuine (if economically negligible) signal. If the null distribution is centred above 0.50 - even at 0.52 - leakage remains in the features or the preprocessing, and the walk-forward AUC must be discounted by that amount.
Corroborating checks, in priority order:
- Univariate AUC per feature. Any single feature with AUC above ~0.58 on its own is a suspect, not a discovery. The index-weight and EPS features should be re-derived point-in-time and the test rerun without them.
- Feature-importance concentration: if one feature carries most of the gain, the model is a proxy for that leak.
- Stability of IC across sub-periods and across the size spectrum. Real signals decay smoothly; leaks often show implausibly strong performance in the earliest years and in the least liquid names.
- Refit preprocessing inside each training fold and confirm the AUC does not move - if it moves a lot, preprocessing leakage was material.
Only after the permutation null sits at 0.50 and the point-in-time rebuild is complete is the 0.54 number informative enough to compare against a cost hurdle.