A Caveat On Backtesting Caveats

Ben Carlson at Ritholtz Asset Management “Developing & Backtesting Systematic Trading Strategies,” is laying out priorities before you write one line of code.

It is important to understand what you are trying to achieve before you set out to achieve it.

Without a set of clearly defined goals there is great potential to accept a strategy or backtest that is really incompatible with the (previously unstated) business objectives and constraints. Worse, it can lead to adjusting your goal to try to follow the backtest, which can culminate in all sorts of bad decision making, and also increases the probability of erroneously accepting an overfitted backtest.

The sad reality is that backtesting is abused, sometimes naively but intentionally as well. The worst offenders are the charlatans who use flimsy backtests to hawk questionable investment strategies and products.

The good news is that an well-constructed backtest is enormously valuable, if only because it forces you to analyze your assumptions by writing and review the rules via computer code. Nearly every coding project I’ve worked on delivered at least one “ah-ha!” moment of deeper clarity about the pros—and cons—of a strategy under scrutiny. In fact, every strategy deserves to be backtested so that the investor has a clearer understanding of the mechanics. Having coded countless strategies over the years (mostly in R ), I’ve found the work to be profoundly enlightening, even when the results more or less fall in line with expectations.

There’s an assumption that only complicated, short-term trading strategies are fair game for backtesting. But relatively conservative portfolio strategies with medium- and long-term horizons can benefit from backtesting analysis by shedding light on the inner workings of interactions between, say, a dozen ETFs that are rebalanced periodically. You may or may not adjust the strategy after this analysis, but the process of quantifying assumptions and reviewing how the risk and return metrics stack up is almost always an edifying effort. The insight may be subtle—recognizing that a fund offers far less diversification benefits than assumed, for instance. Or maybe the results reaffirm your expectations. Another possibility: a backtest reveals a previously overlooked weakness in the strategy.

Backtesting as an educational tool, in other words, is arguably where the most value-added insight lies. By contrast, hitting a home run by uncovering a previously unknown investing system that generates high returns with low risk is the equivalent of winning the lottery—it’s possible, but unlikely.

The simple truth is that there’s no avoiding backtests. Even a buy-and-hold strategy is based on a backtest, albeit a simple one.

Get The News You Want

Read market moving news with a personalized feed of stocks you care about.

Get The App

We shouldn’t whitewash the caveats in backtesting, but that’s no reason to assume the worst. The potential for abuse is vast, but so is the opportunity to inform and illuminate.

Written By: James Picerno