Why constantly optimizing fails to work. Proof.

 

http://gifmaker.me/PlayFrameAnimation.php?folder=20140803173NE4ZBodV7NHtAUIIdUoVA

In the link posted I created a gif image of 3d optimization graphs.   This is an EA trading EURUSD.  there are 25 optimization passes/ settings combinations. Starting from january 2012 for every month each graph frame in the gif represents a 3 month test.  You can clearly see the problem of always preparing EA settings for the next month setting based on the last few months.  

The setting combination that performs the best had a very small chance to be the best and is very unlikely to be the best again next month.  

This problem keeps traders always chasing, and never catching the best profit.  

My advice:   Use many symbols with the same settings.  Run an "all symbols in market watch test" making sure only the symbols you'd want to trade are in the market watch window. If the cumulative profit/loss on all the symbols is positive, use those symbols settings and stick with it for years.  As favorable behaviors on some symbols disappears, they may appear in other symbols.  The point is not to overfit a system to any specific symbol, timeframe, or period in the past, it should generally work across all markets.

What are your thoughts? 

 
johnnypasado:

http://gifmaker.me/PlayFrameAnimation.php?folder=20140803173NE4ZBodV7NHtAUIIdUoVA

In the link posted I created a gif image of 3d optimization graphs.   This is an EA trading EURUSD.  there are 25 optimization passes/ settings combinations. Starting from january 2012 for every month each graph frame in the gif represents a 3 month test.  You can clearly see the problem of always preparing EA settings for the next month setting based on the last few months.  

The setting combination that performs the best had a very small chance to be the best and is very unlikely to be the best again next month.  

This problem keeps traders always chasing, and never catching the best profit.  

My advice:   Use many symbols with the same settings.  Run an "all symbols in market watch test" making sure only the symbols you'd want to trade are in the market watch window. If the cumulative profit/loss on all the symbols is positive, use those symbols settings and stick with it for years.  As favorable behaviors on some symbols disappears, they may appear in other symbols.  The point is not to overfit a system to any specific symbol, timeframe, or period in the past, it should generally work across all markets.

What are your thoughts? 

I believe every symbol got their own movement range. So EA might work on this and not that. The design of the EA plays a part too.
 
johnnypasado:

My advice:   Use many symbols with the same settings.  Run an "all symbols in market watch test"...

What are your thoughts? 

I agree with you.

Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.

Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.

An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).

With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.

[PROPOSAL] Use of a Quotes Synthetizer for deeper testing of EAs
[PROPOSAL] Use of a Quotes Synthetizer for deeper testing of EAs
  • www.mql5.com
Anyone interested to develop this idea and share possible solutions is welcome. - - Category: general
 
oddpip:

I agree with you.

Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.

Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.

An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).

With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.

Hello 

Interesting idea. But the idea already exists between statistician and econometrician. Also the limitation of the method is well known too.

One thing you should be careful is that why do you think sine wave or other Synthetic data represents the property or characteristics of historical price or future price. If they don't, it is almost no point to go though this kind of extended simulation because they won't reveal anything.

One thing  we know for sure is historical variation. You might tackle this using Monte Carlos simulation approach. Variation is only one property of our price series among many other properties. So it is far from being perfection and satisfaction in my view. 

I hope this comment is useful for your future development work and good luck with them. 

Kind regards.

 

It is difficult to generalize that constant optimization fails because simply many profitable system is based on constant optimization. Even manual traders, somehow they will constantly optimize their trading strategy.

This will be done in the manner they look at historical performance of their trading strategy either monthly or quarterly.

Constant optimization is good when you do it to avoid the same mistake you made in the past. However, constant optimization is bad if it tricks you to see only optimistic future which is not really existing.

So use constant optimization for right context.

Kind regards.

 

Sorry to anyone wanting to view the GIF at this time, apparently the file is only stored on their server for a few days.

doshur:
I believe every symbol got their own movement range. So EA might work on this and not that. The design of the EA plays a part too.

The problem is that every symbols movement range/ oscillation / forces / behaviors, whatever you want to call it, changes.  For example December's EURUSD 2015 behavior may be like AUDCAD's March 2001 behavior (theoretical not actual, just making a point) or what not- the only way I would trust an EA to work in the future is if it can work on over 50% of the symbols I want to use with the same settings.  Because you are right, each symbol has a different behavior- but their behavior is not exclusive even to it's own symbol, month to month the variation can be crazy, year to year also.  We can never trust the "best setting" from an optimization to work tomorrow.  We must subject an EA, trying one pair of settings on all symbols because over any unit of time all symbols exhibit different behaviors.  

 

oddpip:

I agree with you.

Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.

Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.

An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).

With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.

Exactly, one single configuration, all symbols really is the beset stress test, thanks for agreeing with me ;)  and i think a white noise test with random small medium large and massive price swings is a great stress test as well. I actually have a fantastic idea about how to implement this type of test, but right now I'm spending all my time working and studying in a summer session class at school and have no time for coding for the rest of the month.  If anyone wants to try to code the idea private message me. 

FinanceEngineer:

It is difficult to generalize that constant optimization fails because simply many profitable system is based on constant optimization. Even manual traders, somehow they will constantly optimize their trading strategy.

This will be done in the manner they look at historical performance of their trading strategy either monthly or quarterly.

Constant optimization is good when you do it to avoid the same mistake you made in the past. However, constant optimization is bad if it tricks you to see only optimistic future which is not really existing.

So use constant optimization for right context.

Kind regards.

The highlighted part:  Absolutely agree with this.  And it can be hard for traders to understand the difference between a bad trading plan and a good trading plan's bad month or two. 

Say if a breakout strategy worked on a symbol last year, and all of a sudden it becomes fake-out season on the instrument, when do (or should) we decide to adjust the algorithm to trade against fake-outs? It is possible that by the time we notice and adjust the algorithm, that fake-out season will end and normal breakouts occur again- this is all theoretical and overly simplified problems, but even so are real problems none-the-less.

When using EA's I wouldn't ever pick the one that says "use only on XY symbol with ABC settings", it either means 1) it does not work on many symbols or 2) the programmer didn't care to test it enough so why should I trust the code in the EA?   Anyway, that's why I learned to program in the first place :)  I was just a ballroom dance teacher when I first got my feet wet trading ~10 years ago, sometime later I bought Fap Turbo and was thoroughly disappointed when it lost thousands of my dollars- and my hatred for those guys increased every day, they never answered my emails, not once, I tried other EAs too with the same bad results until I broke and decided to learn to program.

Before I ramble on with my whole life's story I'll just finish now and say thanks for your replies guys, I'm sorry the GIF isn't there anymore... I'll try to post it here as an inserted image, if it works you'll see a gif with changing images, if not it will just be a picture of the first frame.

 

Ok before "Add coment" i can see the gif changing images, lets see if it works after posting... 

 

Nice. that's better.

If someone want's to do this it would be awesome to see ~260 tests starting at almost every day of the week (M-F) for one year, each of which are one year tests.  I don't have time for a while.  I'm guessing the green islands might gradually instead of dramatically shift as in the last image which was starting every month for test periods of 3 months.

 
johnnypasado:
About your gif, is slow to load because it weighs 1.69 Mb.

You can optimize GIF animations, also online.

Try this, for example: http://ezgif.com/optimize

this is reduced to 577 Kb:


GIF animation online optimization
  • ezgif.com
This tool is designed to shrink gif file size by reducing number of colors in each frame. Each gif frame can use up to unique 256 colors, and by reducing this number, you can achieve smaller file size. It makes multiple variations of input image, and you can choose the one that has best size/quality ratio for your needs. Lossy GIF Compresses...
 
doshur:
I believe every symbol got their own movement range. So EA might work on this and not that. The design of the EA plays a part too.
It looks everyone missed the highlighted part, which is very important. For example, if EA is designed in such a way that all parameters are normalized and should be entered as symbols characteristic metrics (for example, fraction of variation instead of points), then the EA sould apparently work on all symbols with equal performance. So, when you demonstrate an animation of an abstract EA which lose, without any info about the EA itself and what has been optimized, this is just not correct. Also please slow down your animations (if you plan to post more), because it's very hard to see something when frames flicker so fast.
 
And one more thing to mention. Yes, everything changes, but the market has a large mass, so it can't change fast (in a little time). You just need to choose this proper "little time". In fact this is a forward period/backward period ratio choosing problem. If I understand correctly, you mentioned that used 3 months for backward optimization and then 1 month for forward test. This is obviously not the "little" enough. You should consider optimizing on a year or more to get stable results on next 1 month. This is just a matter of meta-optimization, which you did not perform at all.
 
johnnypasado:

Sorry to anyone wanting to view the GIF at this time, apparently the file is only stored on their server for a few days.

The problem is that every symbols movement range/ oscillation / forces / behaviors, whatever you want to call it, changes.  For example December's EURUSD 2015 behavior may be like AUDCAD's March 2001 behavior (theoretical not actual, just making a point) or what not- the only way I would trust an EA to work in the future is if it can work on over 50% of the symbols I want to use with the same settings.  Because you are right, each symbol has a different behavior- but their behavior is not exclusive even to it's own symbol, month to month the variation can be crazy, year to year also.  We can never trust the "best setting" from an optimization to work tomorrow.  We must subject an EA, trying one pair of settings on all symbols because over any unit of time all symbols exhibit different behaviors.  

 

Exactly, one single configuration, all symbols really is the beset stress test, thanks for agreeing with me ;)  and i think a white noise test with random small medium large and massive price swings is a great stress test as well. I actually have a fantastic idea about how to implement this type of test, but right now I'm spending all my time working and studying in a summer session class at school and have no time for coding for the rest of the month.  If anyone wants to try to code the idea private message me. 

The highlighted part:  Absolutely agree with this.  And it can be hard for traders to understand the difference between a bad trading plan and a good trading plan's bad month or two. 

Say if a breakout strategy worked on a symbol last year, and all of a sudden it becomes fake-out season on the instrument, when do (or should) we decide to adjust the algorithm to trade against fake-outs? It is possible that by the time we notice and adjust the algorithm, that fake-out season will end and normal breakouts occur again- this is all theoretical and overly simplified problems, but even so are real problems none-the-less.

When using EA's I wouldn't ever pick the one that says "use only on XY symbol with ABC settings", it either means 1) it does not work on many symbols or 2) the programmer didn't care to test it enough so why should I trust the code in the EA?   Anyway, that's why I learned to program in the first place :)  I was just a ballroom dance teacher when I first got my feet wet trading ~10 years ago, sometime later I bought Fap Turbo and was thoroughly disappointed when it lost thousands of my dollars- and my hatred for those guys increased every day, they never answered my emails, not once, I tried other EAs too with the same bad results until I broke and decided to learn to program.

Before I ramble on with my whole life's story I'll just finish now and say thanks for your replies guys, I'm sorry the GIF isn't there anymore... I'll try to post it here as an inserted image, if it works you'll see a gif with changing images, if not it will just be a picture of the first frame.

 

Ok before "Add coment" i can see the gif changing images, lets see if it works after posting... 

Simply speaking,  model building for trading is as highly uncertain as trading itself.

At every single step, your best bet is to stick with most sound statistical decision. It is because you don't want to under or over estimate your models.

In my view, trading model is very honest, as long as you know what you are building, they will do exactly how you specified no more or no less.

To be honest, constant optimization is not too much issue for model builder. More important thing to discuss is what is the data you work with. (i.e. Stationarity or non statationarity to start with, and many more to consider).  

In my view, econometrics and time series forecasting books are the bible for this. They will tell you most of answer what model builder want to know. 

Good book by reputable Professors in the area.

Forecasting Economic Time Series By Michael Clements, David Hendry

 

Plus I like your opintion on fab turbo. I really support on it. :) :)

Kind regards.