MT5 strategy tester - low number of passes in the forward assessment phase

[Thomas110]

Hi! 

Currently performing a slow complete optimization on my EA, but quite astonished to see that the number of passes covered in the forward phase is much lower than in the optimization phase (Optimization Results).

I assume MT5 only considers the best passes of the optimization phase to assess them in the forward period; is there an official explanation from MetaQuotes about this?

Do you see a way to force MT5 to consider all passes of the optimization phase in the forward phase?

[Alain Verleyen]

Please check and read the documentation before asking on the forum.

https://www.metatrader5.com/en/terminal/help/algotrading/strategy_optimization

[Thomas110]

Alain Verleyen #:

Please check and read the documentation before asking on the forum.

https://www.metatrader5.com/en/terminal/help/algotrading/strategy_optimization

Couldn't originally find it, thank you for the answer.

For those wondering: "The full optimization (slow or fast) of the Expert Advisor is conducted on the back testing period. After that 10% (in the full search) or 25% (in the genetic algorithm) of best runs are selected and then tested on the forward period."

But I can't prevent myself from wondering why the f*** a 10% ratio is imposed for slow complete opts when we are precisely interested to optimize AND assess all possible combinations. An option could have been foreseen, or default could have been 100%.

[Stanislav Korotky]

Thomas110 #:

Couldn't originally find it, thank you for the answer.

For those wondering: "The full optimization (slow or fast) of the Expert Advisor is conducted on the back testing period. After that 10% (in the full search) or 25% (in the genetic algorithm) of best runs are selected and then tested on the forward period."

But I can't prevent myself from wondering why the f*** a 10% ratio is imposed for slow complete opts when we are precisely interested to optimize AND assess all possible combinations. An option could have been foreseen, or default could have been 100%.

I suppose it would be a waste of computational resources. Most oftenly, an optimization's objective is to find a single best result, and its successful forward test would suffice (iff optimization algorithm and its target are well defined). But alone it's not so convincing that your trading strategy is workable in more broader space of settings, and it does not provide an estimation of how results of the best settings are degraded out-of-sample in comparison with other possible settings. Also the additional sub-optimal results may help you to adjust the optimization objective itself. Moreover some people like to pick up several profitable parameter sets (usually uncorrelated as much as possible) to build up a more robust basket of strategies.

That said, the number of the alternative solutions must be countable and competitive - that is not 100%, taking into account that full optimization may have millions of passes and genetics have thousands. This is why MQ decided to take only 10%/25%. I agree - the numbers could be discussed or requested from user. But this is not the only thing which is not customizable in MT5. They have their own priorities for features implementation.

[Thomas110]

Stanislav Korotky #:

I suppose it would be a waste of computational resources. Most oftenly, an optimization's objective is to find a single best result, and its successful forward test would suffice (iff optimization algorithm and its target are well defined). But alone it's not so convincing that your trading strategy is workable in more broader space of settings, and it does not provide an estimation of how results of the best settings are degraded out-of-sample in comparison with other possible settings. Also the additional sub-optimal results may help you to adjust the optimization objective itself. Moreover some people like to pick up several profitable parameter sets (usually uncorrelated as much as possible) to build up a more robust basket of strategies.

That said, the number of the alternative solutions must be countable and competitive - that is not 100%, taking into account that full optimization may have millions of passes and genetics have thousands. This is why MQ decided to take only 10%/25%. I agree - the numbers could be discussed or requested from user. But this is not the only thing which is not customizable in MT5. They have their own priorities for features implementation.

Clearly, I understand some of the arguments, and also believe having a flexible % of covered passes in the forward phase is a must.

Here are the reasons:

• if you have already performed a fast gen optimization, you might wish to be reassured and see again the same results (back AND forward) in a slow complete run. We already know that this wouldn't be possible by repeating the exact same fast genetic optimization, given the random nature of the latter, so this 10% rule in the slow complete forward assessment doesn't address this important limitation.
• if you have already performed a fast gen optimization, you might wish to compare your previous result with the best one obtained with a slow complete run, and see how far it was from the global optimum.
• interpretations of the slow complete opt are much more limited if only 10% of passes are covered in the forward phase, and the backtest results alone can't be used.
• isn't it stupid to focus on such a top 10% from the optimization phase, as it could be the part of the passes that might actually be overfitted?

Let's hope MetaQuotes will consider this in a next release.

[Stanislav Korotky]

Thomas110 #:

Clearly, I understand some of the arguments, and also believe having a flexible % of covered passes in the forward phase is a must.

Here are the reasons:

• if you have already performed a fast gen optimization, you might wish to be reassured and see again the same results (back AND forward) in a slow complete run. We already know that this wouldn't be possible by repeating the exact same fast genetic optimization, given the random nature of the latter, so this 10% rule in the slow complete forward assessment doesn't address this important limitation.
• if you have already performed a fast gen optimization, you might wish to compare your previous result with the best one obtained with a slow complete run, and see how far it was from the global optimum.
• interpretations of the slow complete opt are much more limited if only 10% of passes are covered in the forward phase, and the backtest results alone can't be used.
• isn't it stupid to focus on such a top 10% from the optimization phase, as it could be the part of the passes that might actually be overfitted?

Let's hope MetaQuotes will consider this in a next release.

[1] - it sounds like you don't believe that the tester will provide the same results for the same settings in different modes. That would be strange. We don't have stats (though you can calculate it, and I think MQ already did this), but I think that 10% of slow optimizations covers 25% of genetic optimization.

[2] - you can still do this with current procedure, the problem here is not in the limited percentage of forward tests, but it the random nature of genetic optimization, which you mentioned yourself.

[3] - these interpretations would be also limited by enormous number of passes, large info bloat, if the limit was not introduced.

[4] - that's the main and correct question, and it means that the tester should do much more analytical work with back and forward tests automatically, than it does now. The problem with selecting more stable solution is not solved by current approach.