Last week, we reviewed a powerful technique for building confidence in your strategy called in-sample/out-of-sample testing.
There are two main ways to create your sets from your backtest: random sampling or by time.
If you come to trading from a different domain, like chemistry or data science, you’re probably accustomed to creating your out-of-sample set using random sampling.
In trading, that’s wrong – you should always segment your trades by time.
That is, choose a date that puts 80% of the trades in the backtest before that date, and the remaining 20% after that date. (or 70%/30% – people get hung up on the percent, but it doesn’t matter that much).
I have a function in my code that I can call that automatically determines this date and segments the trade set appropriately.
So why is random sampling so inferior for our purposes when it’s perfectly legitimate in other domains?
Every trading day is another sample that traders around the world look at, learn from, and adjust their own strategies based on.
When you choose to segment by random sampling, you’ll have trades in your in-sample set that are more recent than your out-of-sample set.
Because of this, your model will appear to be more optimistic than reality because you’ll be training it on “future data” and testing on previous data.
I was convinced this was an issue several years ago when I had a large backtest and tried it both ways and compared.
Although it was subtle, random sampling yielded a model that looked more optimistic than segmenting by time.
When your process is more optimistic than reality, it’s like you’re wearing rose colored glasses – you’re setting yourself up for disappointment later.
Tomorrow, I’ll go over one of the main underlying reasons traders get stuck when they consider curve-fitting.
-Dave
