backtesting - consistency and reliability

New comment
 

I have built an expert advisor using a set of candlestick patterns combined with a simple set of indicators. The indicators are used to decide the trend and then this is applied generically across any of the candlestick triggers to decide whether to trade.

The EA runs on the 4 hour chart.

So the results are good for 2010-2019 (overall PF=2.05, 700 trades) and also for 2020 (PF=1.71, 52 trades), with typical profit factors between 1.6 and 3.0, with draw down of under 20%.

The problem is, when I run against 2021 data, it's a disaster - profit factor 0.59, losing money almost every month.

Of course, it sounds like a typical "curve-fitting" scenario. But I have been careful to try to avoid this. The candlestick indicators are very simple and I apply the other trend indicators to all candles consistently - there's no "fine tuning" of indicators for each candle which would lead to a result highly optimised for historical data.

And 2020 - which was not part of the optimisation - gives a good result.

So does anyone have any idea as to why my backtest results are so out of line with 2021?

Should I ditch the whole thing? Or is it possible that 2021 is just a "bad run"?


Thanks

 
Colin Gilbert:

I have built an expert advisor using a set of candlestick patterns combined with a simple set of indicators. The indicators are used to decide the trend and then this is applied generically across any of the candlestick triggers to decide whether to trade.

The EA runs on the 4 hour chart.

So the results are good for 2010-2019 (overall PF=2.05, 700 trades) and also for 2020 (PF=1.71, 52 trades), with typical profit factors between 1.6 and 3.0, with draw down of under 20%.

The problem is, when I run against 2021 data, it's a disaster - profit factor 0.59, losing money almost every month.

Of course, it sounds like a typical "curve-fitting" scenario. But I have been careful to try to avoid this. The candlestick indicators are very simple and I apply the other trend indicators to all candles consistently - there's no "fine tuning" of indicators for each candle which would lead to a result highly optimised for historical data.

And 2020 - which was not part of the optimisation - gives a good result.

So does anyone have any idea as to why my backtest results are so out of line with 2021?

Should I ditch the whole thing? Or is it possible that 2021 is just a "bad run"?


Thanks

The market always changes and you can not expect your optimization bee good forever. all optimizations have their "shelf life". Some are good only for several days after the optimization period, some for months. You say yours is good a whole year (2020) after the end of the optimization. That's very, very good! Your next step is to make a new optimization including data till this day. And be ready to remake your optimizations after regular periods of time.

 
I dont agree with that… I think strategy tester is just to test you EA  to see if it is running as desired … optimization is just to get an average of your parameters for indicators … to adjust how fast it reacts according to the asset historical movement… BUT NOT to decide how profitable it is … for this you should forward tested for several months … and there is your profit/Drawdown and other statistics 
 

If you're talking forex, 2021 has been pretty bad (at least in my own experience), generally low volume and lots of unexpected reversals

Have you checked the 2010-2019 years individually? I think there were some bad years in there too

New comment