Optimisation and Out-of-Sample Testing. - page 6

New comment
 
Mathemat:
Vita, good point. But I would say that generalization ability is not only a property of EA, but also a property of learning algorithm, which must correctly identify this ability. The methaquot algorithm does not reveal this ability, it destroys it by overoptimization.

A metaquote algorithm doesn't pick up on this capability, but instead gets chewed up by over-optimization. -- I'd like to see an example.
 

Examples abound. I was unclear. I meant that ill-considered use of optimizer a la "one, two, three!" leads to excessive and unreasonable optimism of graylists. For example, the latest example - 'Is Forex moving against us? Or maybe I don't understand it'. Or 'Trend Advisor (Expert). I'm waiting for the feedback.'. These are all tricks.

About 'destroying at the root': the probability that marginal curvafitting on one data segment will lead to sustained profitability on the real, out-of-sample segment, is extremely low. The optimum is somewhere in the middle between overprofitability and breakevenness, which is inherent in the NS training algorithm.

Tell me please, Vita, does our genetic optimizer in its pure form provide any guarantee of profitability of an Expert Advisor in the future - even if it is super-duper profitable in testing/optimization? I'm not talking about the tester, the tester is just fine.

P.S. Finally found one of the very first mentions of genetic optimizer by MQ: https://www.mql5.com/ru/forum/50805. See Filin's post and Slawa's reply. The answer shows that the optimizer is useful only for the rough estimation of an Expert Advisor's capabilities and nothing more. The majority of graylists use it as the first and the only tool of "total" testing of Expert Advisors. This is an amateurish approach that leads to false illusions. In fact, this tool, intended only for rough estimation, is, by the law of meanness, good only for scrapping obviously bad Expert Advisors in case of obviously negative results - but not for making positive conclusions about the Expert Advisor's value...

 

Mathemat писал (а):

Examples abound. I was unclear. I meant that ill-considered use of optimizer a la "one, two, three!" leads to excessive and unreasonable optimism of graylists. For example, the latest example - 'Is Forex moving against us? Or maybe I don't understand it'. Or 'Trend Advisor (Expert). I'm waiting for the feedback.'. These are all tweaks.

--> :) Got it. However, agree, meta-quotes can't be responsible for 'rash application of optimizer'. I'm in favor of human thinking and optimizer optimizing. I, for example, know the law according to which my Expert Advisor trades. The optimizer gives me a lot of profitable sets of parameters. How it obtains them - "bluntly" with the help of genetics or "cleverly" with the help of a neuronet - it is of no importance to me, because in any case I critically (deliberately, so to speak) look at the results and immediately separate flies from cutlets, because I'm the only one who knows exactly the law by which my Expert Advisor trades.

--> Imagine that we have a "smart" optimizer that can accurately say: "Here's the optimal set of operating parameters. I swear, no adjustment!" or "No, guys, for this EA I can not pick working parameters, and not just fit the curve." Those would be the times! An optimizer certifying the strategies implemented in the EAs! Free access from metaquotes! Great! I really want one.

About "killing at the root": the probability that the marginal curvafitting on one data interval will lead to sustained profitability on the real, out-of-sample interval is extremely low. The optimum is somewhere in the middle between breakeven and superprofitable, which is inherent in the NS learning algorithm.

--> The "optimum is somewhere in the middle" is an intuitive feeling resulting from the fact that the fully optimized "graals" are guaranteed to fail in the future, so one should not optimize to the end, or that "the truth is somewhere near" or something like that. This feeling has nothing to do with reality. The regularity can be optimized to the very limit and only there it will reveal itself in all its glory. No matter how much you twist it, it will be of no use, but the feeling that the optimum is somewhere in the middle can be developed.

Please tell me, Vita, does our genetic optimizer in its pure form provide any guarantee of profitability of an Expert Advisor in the future - even in case of super-duper profitability on the testing/optimization part? I'm not talking about the tester, the tester is just fine.

--> I'm not, and I think it's inappropriate to demand such guarantees from the optimizer's results. See above about "smart", giving guarantees, optimizer.

P.S. Finally found one of the very first mentions of genetic optimizer by MQ:

https://www.mql5.com/ru/forum/50805

. See Filin's post and Slawa's reply. You can see from the answer that the optimizer is only useful for a rough estimate of an Expert Advisor's capabilities and nothing more. The majority of graylists use it as the first and the only tool of "total" testing of Expert Advisors. This is an amateurish approach that leads to false illusions. In fact, such a tool intended only for rough estimation is, by the law of meanness, good enough only for scrapping obviously bad Expert Advisors in case of a clearly negative result - but not for making a positive conclusion about the Expert Advisor's value...

-->"The optimizer is useful only for rough estimation of EA's capabilities" - again, the claim to the optimizer that is unable to accurately estimate the capabilities of an EA, but, in fact, to certify the strategies of an EA. It only remains to create a robot that generates strategies, feed them to the optimizer, get an opinion about their suitability, and go to the guaranteed cutting of money. How do you like this illusion?


 

Illusion is great, but only an illusion. But it's possible to dramatically improve the application of what we have (tester/optimizer) with the means available in MQL4. Otherwise we'll still see super choppers every three days with only fitting results, pretending to be normal testing...

And I don't have any complaints about metaquotes: the optimizer does exactly what it was designed for, i.e. genetic optimization in Expert Advisor parameter space, and in fact it is actually a curve fitting.

I have a suggestion for the tester: it would be nice to have a standard function like test() in which all necessary parameters would be clearly specified, including the name of the history file on which everything is tested. Rigid binding of a file name to its contents is not a flexible enough solution.

 
leonid553:

Good afternoon to all.

After optimizing an EA, we often have to nerd out-of-sample more than a dozen sets of parameters suggested by the optimizer.

I have an idea of optimizing Expert Advisors outside the sample. Suppose we "charged" the Expert Advisor with optimization by a number of parameters. We set a date. For example, from 1 Jan. 2006 till January 1, 2007.

We have received several thousand Expert Advisors. After that, we save the page with the OPTIMIZATION RESULTS as a separate file. Next, we set the following history period for optimization, i.e. we add a month or two, or as many as we need.

In our case, we set for example from 1 Jan. 2007 to June 1, 2007. And again we use optimization. To be more exact, it won't be quite optimization. The optimizer should not take parameters in EXPERT'S PROPERTIES, but re-select sets of parameters from the file we saved after the first optimization. After this second optimization, we are left with only those vAreAs that yielded a profit outside the sample!

The result, ideally, is that we get the "ideal parameters" to work with and test online later on!

I think this will be a useful addition to the mt4 tester. Probably, and most likely, it is already implemented by someone somewhere. If anyone knows, please share the link!

I, due to my modest knowledge, cannot figure out how to go about implementing the idea in practice.

Surely there is already a practical implementation of this algorithm ... On the forum I found only its derivatives... For instance, 'How to implement your optimization criterion'...

I want to share my solution to this problem....

Let's prepare an EA... Let's add external parameters...

extern int    Counter      = 1;        // Счетчик подсчета количества оптимизаций. Если равен 0, то обычная оптимизация...
extern string Filename     = "";       // Имя файла с оптимизируемыми параметрами

int    FilePtr=0;                      //  Положение файлового указателя

Into function init() insert the following block....

   if(IsOptimization() && Counter>0)
   {
// Для ускорения процесса будем поддерживать сквозной указатель файловой позиции
// с помощью глобальной переменной FilePtr 
// На первом проходе инициализируем её нулём.
      if (GlobalVariableCheck("FilePtr")==false || Counter == 1) 
      {
         FilePtr = 0; 
         GlobalVariableSet("FilePtr",0); 
      } 
      else 
      {
         FilePtr = GlobalVariableGet("FilePtr"); 
      }

      int handle=FileOpen(Filename,FILE_CSV|FILE_READ,';');
      FileSeek(handle,FilePtr, SEEK_SET);
      Parametr1=StrToInteger(FileReadString(handle));
      Parametr2=StrToInteger(FileReadString(handle));
      Parametr3=StrToInteger(FileReadString(handle));

      FilePtr = FileTell(handle); 
      GlobalVariableSet("FilePtr",FilePtr);
      
      FileClose(handle); 
   }

Parametr1, Parametr2, Parametr3 - external parameters, which should be optimized....

That's all ...

 
kharko:

Surely there is already a practical implementation of this algorithm... On the forum I found only its derivatives... For example, 'How to implement your optimization criterion'...

I want to share my solution to this problem....

Take a look at articles by Nikolay Kositsin, e.g. 'Experts Based on Popular Trading Systems and Trading Robot Optimization Alchemy (Continued)'.

 

How does it work?

At time interval A we run the usual optimization of parameters (Counter=0) ...

We transfer the results to Excel... Now our task is to create a file with optimized parameters and save it to the directory ...\tester\files

Select columns with our parameters in Excel, copy and paste them into Word or Notepad as unformatted text...

In Wordboard or Notepad convert each line to the form: value1;value2;value3.

Save it in the directory ...\tester\files

If you're not too lazy, you can write a macro to perform the above operations on the fly...

Now we can run optimization on time slot B... Now the optimization parametre will be Counter... Specify the maximum value (number of rows in the list)...

That's it, the problem is solved... Good luck...

 
Vinin:

Check out articles by Nikolay Kositsin, such as this one 'Expert Advisors based on popular trading systems and the alchemy of trading robot optimisation (Continued)'.

I read this article... I think my variant is simpler and more convenient.... and most importantly it's universal...

The above implementation coincides completely with the branch author's wish...

 
A real-life example of optimisation... See attachment
Files:
sample.zip  1201 kb
 

Thank you, kharko for the solution. I'll try to use it!

123456789101112
New comment