There's probably a more accurate term for this, but here's a story:
What if I told you that I have a methodology that beats the S&P 500 eighty-percent of the time? And what if I added that I backtested it for the past 25 years? You'd be impressed right, given that only 10% of mutual funds beat the S&P 500 and that most backtests are 10 years.
But I'd be withholding some crucial information: this is the third methodology that I tried.
The first one I tried was to pick stocks according to the release date of game-changing products. For example, investing in Apple when the iPhone or iPod came out (Note: this assumes you can identify a game-changing product before the rest of the investors can, which is nowhere near a safe assumption). However, when I backtested this, the results were a wash.
The second methodology I tried involved investing in Fast Company's list of Innovative companies. The idea is that innovative companies have to keep growing. However, the results here were a wash too.
So now, we come to my third methodology (which I won't disclose right now), and aha! Not only does it have interesting narrative logic, it backtests profoundly well! So give me all your money!
The moral of the story is that given enough attempts, you can find a formula/methodology/algorithm/system that backtests incredibly well against benchmarks. Likewise, every year, 5-10% of mutual funds are going to have amazing historicals. But in a year, there will be a different list of funds, and a year later, a another list.
At best, backtesting should be a sanity check. If your formula doesn't backtest well, then it's probably not valid. But if it does, that should just be the beginning of your analysis.
