For the next few days, I’ll be answering an excellent set of questions from a Strategy Cruncher user named Chris B., who agreed to share these with the list.
Here’s the first question:
Chris B.
I’ve been playing with the Strategy Cruncher with an intraday system I’m running that I’ve backtested using RealTest.
- The system trades SPY intraday, long and short.
- I have data from 2001 through the present.
- In-sample is 6/2002 through 12/2014.
- Out-of-sample is 1/2015 through today.
- I separated long and short trades into two CSVs.
I ran the in-sample set of the short side of the strategy through the cruncher with the aggression factor set at 33 with 1456 trades and this was the best chart.
I noticed all the charts had accelerating equity curves during 2008 so I asked myself how might SC suggest values without that year of trades. I ran the short trades through SC again without 2008, which produced this chart:
Both filter values for Dojiness clearly improve the system in terms of CAR and MDD with about 40% fewer trades.
Does eliminating that period of trades make sense, or is that just overfitting?
Dave:
This is a great question, Chris. I would not consider this overfitting for a few reasons.
It would be one thing if the period you remove from the backtest is worse than the overall backtest, but this period improves the strategy. That difference is significant – if the period made the strategy worse overall and you removed it for testing purposes, then I would say that IS more likely to be curve-fitting.
I also consider the period itself. Is it a significant event that you can identify in a way that makes sense? Yes, this is the 2008 financial crisis.
How likely is this type of event to recur in the future? Sure, you can imagine a scenario where it might occur again, but it’s not very likely!
This strategy should be optimized for the “normal” periods, which are the vast majority of the time covered by Chris’ backtest.
The equity curve for the rule the cruncher found looks much smoother by looking at just the “normal” periods.
I don’t create strategies that wait around for these black swan-type events that very likely will never come about again. And if it does recur, there’s certainly no guarantee they’ll behave the same way.
I think this is a great example of what I discussed in the Curve Fitting series a few weeks ago. Traders get stuck with this type of decision for fear of the curve-fitting boogeyman.
Does this mean you can exclude whatever periods you want to make your strategy look better? Of course not.
But I think it’s totally legitimate to remove this period in exactly the way Chris did.
I’ll tackle another one of his questions tomorrow. Thanks, Chris!
-Dave
P.S. Would you like to improve your strategy by finding ways to make the same or more profit with way fewer trades without curve-fitting? Lots of serious traders are using the Strategy Cruncher to create their own trading strategies. If you have a backtest, you should be crunching.
