In response to my initial question when I introduced the topic of curve-fitting, fellow list member and trader Mateo shared his experience when he recognized he had curve-fitted and what he did to verify and fix it (sharing with permission)
Mateo:
I lost money a few years ago on SPY, which I attribute to curve fitting, but I wouldn’t have called it that back then. I had a great looking and producing backtest on SPY. I placed a long call that lost, and so I had an idea: test it on similar tickers! It was amazing on SPY but completely failed on VOO and SPX. I realized that I should get substantially similar results on something so correlated. I found I had filtered out all the bad trades too exactly.
I’m super interested to read your curve fitting insights because you often make me think and eventually change my perspective on market topics.
Dave:
This is exactly the kind of thing I was looking for. So, how long did it take you to figure out that you had curve fitted? And how big was the loss relative to what you were doing at the time?
Mateo:
I figured it out very quickly once I had the idea to try it on other symbols. I only recently learned the word “robustness testing,” but I’d say within a couple of hours. In this case, it was a small loss, thankfully.
Thanks so much to Mateo for sharing his experience!
He highlights a great way to quickly recognize curve-fitting in his strategy, and he recognized quickly (a couple hours!) what was going on well before significant losses mounted.
If your strategy is well-designed and predictive, it should generalize to other tickers and markets.
If it doesn’t (as Mateo discovered), it should give you pause. It doesn’t necessarily mean you should abandon your idea, though.
Next time, I’ll discuss ways to try to improve his strategy without completely giving up on it.
-Dave
