Genetic algorithm blocking further optimizing without being asked for
So my question is: how can I get around this? How can I tell MT5 I don't want the GA to be used no matter what it thinks is the best for me?
You can't.
- If the total number of optimization steps exceeds 1,000,000 in a 32-bit system or 100,000,000 in a 64-bit system, the genetic optimization mode starts automatically.
Source: https://www.metatrader5.com/en/terminal/help/algotrading/optimization_types
About your real problem:
I have the same problem (you know my work, so you know I do... :D) since 2015 and found no soluction so far.
Actually I couldn't find somone that understand it till now (you're the first).
Here is my original post:
- www.metatrader5.com
About automatic switch to GA, 100,000,000 passes with 5 seconds for a pass would be insane, don't you think ?
About your main issue, there is no built-in solution. Using INIT_PARAMETERS_INCORRECT need to be used sparsely to avoid discarding too much passes, which would wreck the GA algorithm. Metaquotes CEO roughly estimated to no more than 25% of combinations discarded (and rather recommend to not use INIT_PARAMETERS_INCORRECT at all). It probably doesn't fit for your problem anyway. Your 'problem' doesn't seem linear so you probably need to forget the idea to use 1 optimization process for all your parameters.
You need a custom solution : you split your problem in X sub-problems, use an external script (whatever language you like) to launch MT5 in optimization mode automatically for each sub-problem, then combine the results. Personally I don't think it's worth the pain as anyway your optimization will lead to curve fitting, but if you believe it's worth it, that's the way to go.
N.B: Actually you could even use MT5 to launch other MT5 running optimizations.
You can't.
- If the total number of optimization steps exceeds 1,000,000 in a 32-bit system or 100,000,000 in a 64-bit system, the genetic optimization mode starts automatically.
Source: https://www.metatrader5.com/en/terminal/help/algotrading/optimization_types
About your real problem:
I have the same problem (you know my work, so you know I do... :D) since 2015 and found no soluction so far.
Actually I couldn't find somone that understand it till now (you're the first).
Here is my original post:
Well that's too bad :( Even though understandable, I think MetaQuotes should have given the last word to its clients if the GA should or not automatically be switch on. Thanks for the reply; at least now I know up till how much iteration I can do each time :)
About automatic switch to GA, 100,000,000 passes with 5 seconds for a pass would be insane, don't you think ?
About your main issue, there is no built-in solution. Using INIT_PARAMETERS_INCORRECT need to be used sparsely to avoid discarding too much passes, which would wreck the GA algorithm. Metaquotes CEO roughly estimated to no more than 25% of combinations discarded (and rather recommend to not use INIT_PARAMETERS_INCORRECT at all). It probably doesn't fit for your problem anyway. Your 'problem' doesn't seem linear so you probably need to forget the idea to use 1 optimization process for all your parameters.
You need a custom solution : you split your problem in X sub-problems, use an external script (whatever language you like) to launch MT5 in optimization mode automatically for each sub-problem, then combine the results. Personally I don't think it's worth the pain as anyway your optimization will lead to curve fitting, but if you believe it's worth it, that's the way to go.
N.B: Actually you could even use MT5 to launch other MT5 running optimizations.
Well since many if not most of the possible combinations lead to be ignored with the call to INIT_PARAMETERS_INCORRECT, I don't mind the 100 million passes with 5 seconds "each". In the dead cases, MT5's log is telling is taking less then 1 ms to kill the invalid passes. Furthermore I only have 16 3.6GHz processors to do the work, but what if I had 50+ like some pros have? In that case I could put one billion passes without a problem and the automatic switch to GA would only be in my way.
Regarding using INIT_PARAMETERS_INCORRECT, yeah I noticed it. I tested in one of the cases and concluded I'ld have do discard their use completely in order to avoid messing with the GA. Still, as I sad, I don't trust this algorithm to do the work anyway.
About your custom script idea, I remember reading about it somewhere like that; create some .set with some optimization parameters configured and then run a scrip that makes MT5 test the sequence of .set. Only problem is I don't see much gain in work from doing it manually parameter group by group.
Finally, even though I'm almost certain I'm not going to use it, I'm curious about your last "N.B.": are you talking about MT5 scripts?
And thanks for the reply!
I think MetaQuotes should have given the last word to its clients if the GA should or not automatically be switch on.
I guess it's related to the max ID it can handle because the limit depends on the CPU architecture
Not manually, automatically.
You make a script that run one set, wait till it finnish than run the next.
Have you seen my "Miner"? It's what it does.
Check this:
https://www.metatrader5.com/en/terminal/help/start_advanced/start
- www.metatrader5.com
Not manually, automatically.
I'm sorry, poor English :D I meant: compared to doing it manually, I can't see much gain from this approach. Manualy:
- Configure a group parameter options which are compatible in how they use the parameters (e.g. use the first 3 stop loss options which all don't require configuring the indicator period) in MT5
- Run
- Catch the data (Print-screen, etc.) and analyse
- Repeat the process again and again till all possibilites are covered
- Compare all the results to select which is the best, most consistent combination of parameters.
The script approach:
- Configure one group parameter to optimize and save the preset as a .set file.
- Repeat step 1 till you have as many as .set files to cover all possible parameter configurations.
- Create a script that will make MT5 run all the .set.
- Run
- Collect the results data somehow (possibly requiring saving them in some CSV file since each new .set run will erase MT5's table of results. In this case, one more step is required: create the save-to-csv mechanism)
Well that's too bad :( Even though understandable, I think MetaQuotes should have given the last word to its clients if the GA should or not automatically be switch on. Thanks for the reply; at least now I know up till how much iteration I can do each time :)
Well since many if not most of the possible combinations lead to be ignored with the call to INIT_PARAMETERS_INCORRECT, I don't mind the 100 million passes with 5 seconds "each". In the dead cases, MT5's log is telling is taking less then 1 ms to kill the invalid passes. Furthermore I only have 16 3.6GHz processors to do the work, but what if I had 50+ like some pros have? In that case I could put one billion passes without a problem and the automatic switch to GA would only be in my way.
Regarding using INIT_PARAMETERS_INCORRECT, yeah I noticed it. I tested in one of the cases and concluded I'ld have do discard their use completely in order to avoid messing with the GA. Still, as I sad, I don't trust this algorithm to do the work anyway.
About your custom script idea, I remember reading about it somewhere like that; create some .set with some optimization parameters configured and then run a scrip that makes MT5 test the sequence of .set. Only problem is I don't see much gain in work from doing it manually parameter group by group.
Finally, even though I'm almost certain I'm not going to use it, I'm curious about your last "N.B.": are you talking about MT5 scripts?
And thanks for the reply!
What you can do with a VBA or Python or whatever script, you can also do it with mql5. It's all what I mean.
The script approach:
- Configure one group parameter to optimize and save the preset as a .set file.
- Repeat step 1 till you have as many as .set files to cover all possible parameter configurations.
- Create a script that will make MT5 run all the .set.
- Run
- Collect the results data somehow (possibly requiring saving them in some CSV file since each new .set run will erase MT5's table of results. In this case, one more step is required: create the save-to-csv mechanism)
I guess it's related to the max ID it can handle because the limit depends on the CPU architecture
I don't think so, it's related to the cloud. They don't want the network flooded by billions of passes, done intentionally or not.
In the past, I opened a ticket to ask them to enable any number of passes without falling on GA when running locally. I didn't get a reply or they say no (don't remember exactly).
Not manually, automatically.
You make a script that run one set, wait till it finnish than run the next.
Hi,
I'm currently facing the same problem reported in the MQL4 questions here and at the end of here. Nevertheless I'll describe it in case it might help.
I have an EA I want to optimize and it has many configuration parameters, e.g.: 11 different types of stop loss strategies. Each such parameters may need one or more configuration option. For example, ATR stop loss needs not only the multiplier value, but the value of the indicator's period as well. But some of such parameters may be useless for other options. Example: the ATR stop loss needs the input on indicator period, but stop loss based on fixed distance of ticks from position start doesn't. Well I want to make "full optimization", not having to stop hundreds of times to configure the .set to the exact same situation it neededs. That, though, will create a lot of unecessary passes. For example, when testing for fix tick distance stop loss, all the many variances of the input parameter of indicator period will lead to the same exact result since this parameters is not being taking into account, leading to lost in time. That brings the big question: how to avoid unnecessary passes?
Two main ways have been proposed in my readings: use the Genetic Algorithm or validate which combination of inputs are valid and use
in OnInit() function when the useless combination of parameters are being tested by the Optimizer.
Well I want to use the second option since I don't trust the Genetic Algorithm to do this work. Problem is that even if I select "Complete Slow Algorithm" instead of GA to do the tests, if too many cases of invalid parameters happen at the start, the GA, which entered in scene without me having asked for it, blocks further development after 512 attemps:
In my point of view, the main cause lies here:
Since I've asked to optimize lots of possible results, the GA has automatically been activated even though I strictly sad I didn't want to use it, rather trusting in my own method of parameter evaluation to speed up the optimization.
So my question is: how can I get around this? How can I tell MT5 I don't want the GA to be used no matter what it thinks is the best for me? And if that is not possible, how can I make the GA continue to optimize even when I configured the testing in such a way many combinations of parameters are going to be invalid?
This is how I got around the issue you are having without using INIT_PARAMETERS_INCORRECT and potentially affecting the outcome of my Genetic backtest or by resorting to creating a set file for every possible viable combination of my input parameters and writing another program to run backtesting on each of those set files individually as previously reported as potential viable solutions to the issue at hand.
1) separate your input parameters which you'd like to apply a kill pass to into different groups. Lets say in one of those groups you have 5 or 6 variables, each of which can have two potential inputs in one of those groups.
2) enter each of those variables and their potential inputs in an excel column as shown below:
3) Modify and use the following excel visual basic and run it on the data you just entered to create every combination of those 5 or 6 variables.
Sub ListAllCombinations()
'Updateby Extendoffice
Dim xDRg1, xDRg2, xDRg3, xDRg4, xDRg5, xDRg6 As Range
Dim xRg As Range
Dim xStr As String
Dim xFN1, xFN2, xFN3, xFN4, xFN5, xFN6 As Integer
Dim xSV1, xSV2, xSV3, xSV4, xSV5, xSV6 As String
Set xDRg1 = Range("A2:A3") 'First column data
Set xDRg2 = Range("B2:B3") 'Second column data
Set xDRg3 = Range("C2:C3") 'Third column data
Set xDRg4 = Range("D2:D3") 'Fourth column data
Set xDRg5 = Range("E2:E3") 'Fifth column data
Set xDRg6 = Range("F2:F3") 'Sixth column data
xStr = "-" 'Separator
Set xRg = Range("H2") 'Output cell
For xFN1 = 1 To xDRg1.Count
xSV1 = xDRg1.Item(xFN1).Text
For xFN2 = 1 To xDRg2.Count
xSV2 = xDRg2.Item(xFN2).Text
For xFN3 = 1 To xDRg3.Count
xSV3 = xDRg3.Item(xFN3).Text
For xFN4 = 1 To xDRg4.Count
xSV4 = xDRg4.Item(xFN4).Text
For xFN5 = 1 To xDRg5.Count
xSV5 = xDRg5.Item(xFN5).Text
For xFN6 = 1 To xDRg6.Count
xSV6 = xDRg6.Item(xFN6).Text
xRg.Value = xSV1 & xStr & xSV2 & xStr & xSV3 & xStr & xSV4 & xStr & xSV5 & xStr & xSV6
Set xRg = xRg.Offset(1, 0)
Next
Next
Next
Next
Next
Next
End Sub
You should have a column of all possible combinations of your variables created as shown below.
4) Now using the Text to Columns function of your excel under Data tab, separate all the combinations into individual columns.
5) Using If statements, apply whatever rules you want to eliminate the rows with variables you do not want to run a backtest on (I use formulas such as =If(D2>C2,1,"") on a new column next to my combination of data to place a 1 in front of every combination that is not suitable to my backtest and then using the Data/Filter show every row which has the number 1 in front of it and tehn delete those rows)
6) Merge the columns again using & into a new column. (example, =A2 & B2 & C2 & D2 & E2 & F2 )
7) Run Data/ Remove Duplicates to remove all the duplicates of combined variables and again separate the data into each individual columns.
8) Introduce a new variable in metaeditor for each of your groups of codependent variables, each input of which will correspond to one of the combinations of the 5 or 6 variables in the excel sheet (For example, in my code, I named that variable AutoTrailVar as such:
input int AutoTrailVar = 1; whereas case of Autovariable = 1 would correspond to the combination of my 5 or 6 variables each having a specific input value and case = 2 would correspond to the 6 variables having another set of inputs.
9) to facilitate the metaeditor programing, I combined all the variations of the code in my excel (see column AE and how the "= G & H & I & ... has combined to write the code i would be cutting and pasting into my metaeditor below:
10) add a function to be read on the int Oninit() as shown below:
int OnInit()
{
//---
AdjustTrailParameters();
And the function of AdjustTrailParameters(); in my program's case is simply a combination of cases for my single variable which has been cut and pasted into metaeditor from my excel sheet as shown above:
void AdjustTrailParameters()
{
//---
switch(AutoTrailVar)
{
case 1: AutoTrail_H1_After_RR=0; AutoTrail_M30_After_RR=0; AutoTrail_M15_After_RR=0; AutoTrail_M5_After_RR=1; AutoTrail_M1_After_RR=3; break;
case 2: AutoTrail_H1_After_RR=0; AutoTrail_M30_After_RR=0; AutoTrail_M15_After_RR=0; AutoTrail_M5_After_RR=1; AutoTrail_M1_After_RR=5; break;
case 3: AutoTrail_H1_After_RR=0; AutoTrail_M30_After_RR=0; AutoTrail_M15_After_RR=0; AutoTrail_M5_After_RR=1; AutoTrail_M1_After_RR=6; break;
case 4: AutoTrail_H1_After_RR=0; AutoTrail_M30_After_RR=0; AutoTrail_M15_After_RR=0; AutoTrail_M5_After_RR=1; AutoTrail_M1_After_RR=7; break;
case 5: AutoTrail_H1_After_RR=0; AutoTrail_M30_After_RR=0; AutoTrail_M15_After_RR=0; AutoTrail_M5_After_RR=1; AutoTrail_M1_After_RR=8; break;
...
11) now when I run a backtest and use different variable input for my Single input "AutoTrailVar' (instead of 5 different inputs for each of my codependent variables), I will only have the combination of variables which result in a meaningful backtest outcome as preselected in my excel sheet without the use of Kill pass.
12) you can do the same for any number of codependent groups of variables within your program for which you'd like to filter out a number of passes as per the guidelines described above.
Hope that helps and good luck.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hi,
I'm currently facing the same problem reported in the MQL4 questions here and at the end of here. Nevertheless I'll describe it in case it might help.
I have an EA I want to optimize and it has many configuration parameters, e.g.: 11 different types of stop loss strategies. Each such parameters may need one or more configuration option. For example, ATR stop loss needs not only the multiplier value, but the value of the indicator's period as well. But some of such parameters may be useless for other options. Example: the ATR stop loss needs the input on indicator period, but stop loss based on fixed distance of ticks from position start doesn't. Well I want to make "full optimization", not having to stop hundreds of times to configure the .set to the exact same situation it neededs. That, though, will create a lot of unecessary passes. For example, when testing for fix tick distance stop loss, all the many variances of the input parameter of indicator period will lead to the same exact result since this parameters is not being taking into account, leading to lost in time. That brings the big question: how to avoid unnecessary passes?
Two main ways have been proposed in my readings: use the Genetic Algorithm or validate which combination of inputs are valid and use
in OnInit() function when the useless combination of parameters are being tested by the Optimizer.
Well I want to use the second option since I don't trust the Genetic Algorithm to do this work. Problem is that even if I select "Complete Slow Algorithm" instead of GA to do the tests, if too many cases of invalid parameters happen at the start, the GA, which entered in scene without me having asked for it, blocks further development after 512 attemps:
In my point of view, the main cause lies here:
Since I've asked to optimize lots of possible results, the GA has automatically been activated even though I strictly sad I didn't want to use it, rather trusting in my own method of parameter evaluation to speed up the optimization.
So my question is: how can I get around this? How can I tell MT5 I don't want the GA to be used no matter what it thinks is the best for me? And if that is not possible, how can I make the GA continue to optimize even when I configured the testing in such a way many combinations of parameters are going to be invalid?