The last question from Chris B. (name used with permission), who’s using the Strategy Cruncher to create an intraday system that trades SPY.
Chris B:
Do you have an objective way to measure a filter’s effectiveness in the out of sample period? My observation when comparing the -0.3308 (dojiness) against the baseline results, the improvements in MAR, Expectancy, AvgLoss, and ProfitFactor are basically the same for in and out of sample backtests. That would give me confidence in the filter.
Dave
You’re right, Chris. It’s a great sign that the threshold the cruncher suggested is the same for your in sample and out of sample trade sets.
That should be a big confidence boost that the optimal filter it found should hold up into the future.
If your strategy’s signal is strong enough and you have enough trades, it won’t matter if you do in sample and out of sample testing.
Why? Because the cruncher will give you the same suggestions whether you have it look at the in sample set or the entire backtest.
If the Strategy Cruncher had suggested wildly different values for the optimal filter for Dojiness for in sample and out of sample sets, what would that say about your confidence in the filter?
You should always be thinking about your “path to confidence” with a strategy.
Trading is easy if you’re trading with small size.
It gets more interesting and difficult when your goal with each strategy is to trade it with significant size.
What questions would you need to answer about your strategy to trade it with more size?
What’s stopping you from getting those answers?
-Dave
P.S. When you’re ready, the cruncher is here. The Strategy Cruncher is like having a cheat code for your trading strategy. Here’s what Matt H. said about the cruncher:
“The cruncher suggested a few filters for my strategy. I plugged them in, ran the back test and WOW! 20% improvement in total profit, and with another, a significant reduction in drawdown. I recommend this for anyone truly serious about optimizing their strategy.”
