On Backtest Overfitting

The central problem in quantitative strategy research is distinguishing real signals from noise. Given enough parameters and enough historical data, almost any strategy can be made to look profitable in hindsight. The question is whether that profitability reflects a persistent market pattern or a statistical artifact.

We address this in three ways.

First, we compute deflated Sharpe ratios using the framework from Bailey and Lopez de Prado (2014), which adjusts for the number of strategies tested, return distribution skewness, and kurtosis. A raw Sharpe of 2.0 from 50 independent trials may deflate to 0.8 after correction. The deflated number is the one that matters.

Second, we estimate the Probability of Backtest Overfitting using combinatorial symmetric cross-validation (Bailey et al., 2017). This evaluates whether the in-sample best strategy consistently underperforms out-of-sample across all possible data splits. A PBO above 0.5 means the strategy is more likely overfit than genuine.

Third, we maintain a trial registry that logs every strategy and parameter combination ever tested. This registry feeds directly into the deflation calculations and prevents survivorship bias in our research records. The failures matter as much as the successes.

References

• Bailey, D. H. and Lopez de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality." Journal of Portfolio Management.
• Bailey, D. H., Borwein, J. M., Lopez de Prado, M., and Zhu, Q. J. (2017). "The Probability of Backtest Overfitting." Journal of Computational Finance.