Here’s a detailed explanation of what I’ve referred to as the “peek-a-boo trade” problem in backtesting. (See the More Trades Than You Expected and the responses posts for more background.)
You’re backtesting for new highs, and there are 900 trades in your backtest.
When you analyze the trades list, you notice that filtering out lower values for volume today improves the resulting trade set and removes 300 trades.
So you add a filter in your backtest that excludes new highs with volume today < X.
When you run a new backtest, you expect to find 600 trades, but you find over 700 trades.
Many traders see this, throw their hands up in the air, and assume the backtest software is somehow broken.
But the backtest is working fine – here’s what’s actually going on.
Take SBET yesterday. It was running up all morning. Because your strategy looks for new highs, your original backtest found the first one at 10:30am. (See chart below)
If you apply a minimum volume today filter at 8.2M, the first instance at 10:30am is excluded, but the next one at 10:45am matches your rule set since volume today has increased by then to 9.7M (9.7 > 8.2).
You’ll still have a trade in SBET in your backtest, but the entry time will be later in the day.
You thought you were excluding the SBET trade, but you were simply postponing it in your backtest.
Add these all up in your backtest, and you’ll often end up with several more trades than you expected.
Over the coming days, I’ll share specific ways you can address this issue in your own backtests.
-Dave
