Outliers are a tricky problem with backtesting. It’s not uncommon to see a backtest consisting of nice, consistent profits and reasonable losses but then a tiny handful of trades that are way, way out of the range. Take this backtest which I’m using a scatter plot to visualize which is one of my favorite ways to view a backtest.
There are several thousand trades in this backtest, and 99.8% of them are within a very, very tight range and can be considered “in line” with my underlying premise for this trading strategy.
But look at the tiny sliver of trades that are in the top part of the plot:
Those outsized profits are very much outside of the norm compared to the other trades in the backtest. How you handle these can make a huge difference in your trading results.
For example, if you’re coming up with a model to apply to the strategy and determining which levels to use for certain parameters by an optimization process, including outliers in your training set could yield very different thresholds than excluding them.
An Extreme Backtest with a Huge Outlier
In fact, I remember a rather extreme example of this. I was testing a really promising trading strategy that I was really excited about trading live as soon as I could. The backtest looked great and I had all my filters dialed in (or so I thought). This thing was ready to go live and I was just putting the final touches on what sizing I would use. Then I look at the scatterplot and noticed a single outlier. It was WAY out there, but it was just a single trade. No big deal right? Just out of curiosity, I took one final look at the model but excluding that one outsized trade and… uh-oh. Picture that same scatterplot above, but with a single trade ten times as profitable as the largest trade at the top of the chart.
It turns out that if I included that single outlying trade, one of my predictive variables was strongly positively correlated with profit, but then when you excluded that single trade in the backtest (just that one!) from the training data, that same variable was negatively correlated with profit! Talk about frustrating!
So what do you do in this case – exclude or include the outliers? When I looked closer into this extreme case, this one single trade was accounting for a large portion of the trading profits. It was still profitable without the trade but from my modeling perspective, it was fundamentally a different trading strategy without the trade.
A Gift Disguised as a Problem
As you gain more experience with trading and backtesting, you start to recognize these frustrating conundrums as really great learning opportunities in disguise. They force you to confront your assumptions head-on and dig deeper. Why is this trade in the backtest? Is it fundamentally in line with my original trading thesis about why this strategy works? Or is it just randomly happening to match my entry criteria? The bigger question, though: if this outsized “happy accident” appeared in my backtest, could there be an “unhappy” accident in the future with a very large loss?
This situation forced me to confront these hard questions for this particular strategy, but it also gave me a path toward coming up with a framework for handling this situation going forward.
My Framework for Outliers
Take the original scatterplot:
Notice that in this case, there are several outliers and they’re all profitable. The fact that there are several AND they’re all in the same direction is a good thing. If there were just one that would worry me a bit more.
Take the same backtest, but let’s say that some of those outliers are negative and some are positive.
Now let’s compare the equity curves for the two strategies:
They’re essentially the same. The other thing to note here is that ANY metric you happen to be using to compare strategies would look essentially identical. Profit factor, win rate, CAGR, total profit, equity curve quality, expectancy, Sharpe ratio, Sortino ratio – they all would tell you that the strategies are basically the same.
In general, though, I’d be a lot more worried about this situation. Why? My concern is that maybe my underlying thesis isn’t as strong as I thought it was. There’s a good chance there’s something I’m missing and that’s what would keep me up at night until I solved it.
I would really think hard about whether these negative outliers violate or nullify my original trading thesis about why the trading strategy works. In general, I’d be worried that I didn’t have a deep enough understanding of the trading edge for this particular strategy and that I was just “blindly following the data” on some level, which can be dangerous.
What’s the Big Deal?
You might wonder: why be so concerned? The equity curves are so similar – what’s the big deal about the negative outliers? They’re dwarfed by the winning trades – so much so that they’re barely a blip on the radar when you step back a bit.
It could be a legitimate response when you confront these outliers to still go forward and trade the strategy as-is. But it’s important to think about them and expect them going forward. It’s much better to confront these before going live with a strategy rather than after.
Imagine the difference between knowing what outliers are possible beforehand, versus trading it unknowingly, sizing up significantly over time, and THEN having an outsized losing trade that you weren’t prepared for. That situation could be catastrophic.
