There were a couple of excellent responses to the backtesting puzzle I posted the other day. (Go back and read it first.)
First, Ken H. responded (used with permission) with an explanation that I wasn’t thinking of but is still perfectly valid:
Ken H.
More trades appeared due to freeing up capital by eliminating existing trades that tie up capital for too long. This means the system is generating multiple signals, and it’s important to add ranking criteria to execute on high-quality entry setups/signals. Also, it’s important to evaluate exit criteria and/or holding time for trades that stop performing.
Dave
Nice response, Ken! That isn’t what I had in mind, but that scenario could absolutely cause the situation I described.
Ken H.
Nice challenge, let me try again.
Let’s say the system looks for continuation patterns and buys on new highs and generates the 900 trades.
I’m studying volume and market profile now and I find that continuation patterns are more reliable with above average volume, so applying an above average volume filter narrows down the outcome to 600 trades. When re-running the backtest with an above average volume filter, it produces 750 trades.
The system could be generating those additional/new trades in various other bars because it considers those other bars as new highs due to filtering out bars with low volume ones. I don’t know if there’s a technical term for this phenomenon.
Matt H. responded first, saying it could be look-ahead bias. When I gave him a hint, he nailed it as well:
Matt H.
Well, entering a few bars after the excluded trade entry bar was skipped, but before the excluded trade finished its life cycle. For example, if you have a timestop of 15 bars, and don’t enter on bar 1 because of the new filter is flagging false, but do on bar 4 or 5 because the filter becomes true. You could end up with more entries than you expected.
Congrats to Ken H. and Matt H. for getting the right answer.
I don’t have a great name for this phenomenon, but I’ve used the term “peek-a-boo problem.”
Suggestions welcome!
I’ll show a real-life example that will demonstrate the issue clearly and then ways I deal with this issue in the coming days.
-Dave
