A Momentum Backtest That Survived Everything
A junior analyst presents a cross-sectional momentum strategy tested on 96 months of data. The mean monthly alpha is 0.45% with a t-statistic of 2.10, which she describes as 'significant at the 5% level'. She selected this specification from 40 variants she ran (different lookback windows, rebalancing frequencies and formation periods) and reports only the best. The fund universe comes from a commercial database that contains only funds still operating at the time of the download. Her conclusion is that the strategy should be funded at full size next quarter.
1. Identify the biases in this work and state the direction of each bias.
Read the worked answer
Survivorship bias (upward). The database holds only funds alive today. Dead funds are disproportionately the poor performers, so both the cross-section of returns and the dispersion are distorted; measured alpha is too high.
Backfill bias (upward). Commercial databases usually add a fund only after it has a track record worth reporting, so a fund's best early years enter the sample while its weak years before listing do not.
Data-mining / selection bias (upward). 40 specifications were run and one was reported. Even if none of the signals had any true predictive power, at a 5% test level you would expect about 40 x 0.05 = 2 'significant' results by chance alone.
Look-ahead and specification issues to check. Rebalancing assumptions, whether the formation window overlaps the holding window, and whether the 96 months were also used to choose the 40 variants (in-sample optimisation).
2. Using the reported alpha and t-statistic, back out the implied monthly standard deviation and the implied annualised information ratio, then assess whether t = 2.10 is convincing after accounting for the 40 trials.
Read the worked answer
Implied volatility. With n = 96, the standard error of the mean is s/sqrt(n), so t = alpha_hat / (s / sqrt(96)) = 2.10, alpha_hat = 0.45%.
s = 0.45% x sqrt(96) / 2.10 = 0.45% x 9.798 / 2.10 = 2.10% per month.
Implied information ratio. Monthly IR = 0.45 / 2.10 = 0.214. Annualised = 0.214 x sqrt(12) = 0.74. That is a very strong number for a published-style anomaly, which is itself a warning sign: results this good usually indicate sample problems rather than genuine inefficiency.
Multiple-testing correction. Bonferroni: required two-sided level = 0.05 / 40 = 0.00125, giving a critical |t| of roughly 3.2. The reported 2.10 falls far short. Even adjusting for correlation among the 40 variants (which makes Bonferroni conservative) the hurdle would remain materially above 2.10.
Conclusion. t = 2.10 is the expected maximum of 40 draws from a distribution centred on zero. It is not evidence of skill.
3. What would you require before funding the strategy, and what is the minimum evidence standard you would set?
Read the worked answer
- Rebuild the universe on a point-in-time basis that includes dead funds, and strip the backfill period (drop the first 12-24 months of each fund's record).
- Re-test the 40 variants on the corrected data and require the winner to clear a multiple-testing hurdle (|t| > ~3.2, or a bootstrap / false-discovery-rate test).
- Out-of-sample confirmation on a genuinely held-back period or a different market. In-sample significance on the same 96 months that generated the variants is not out-of-sample.
- Economic story first. A momentum premium needs a risk- or behaviour-based explanation; a parameter combination that happens to work is not a hypothesis.
- Fund at a size where the result is still economically meaningful after turnover costs. The 0.45% monthly gross alpha is small relative to the trading costs implied by rebalancing across many names, so net-of-cost alpha should be recomputed.
Only a strategy that survives corrected data, an out-of-sample period, a multiple-testing hurdle and costs justifies full-size funding.