Not for MT developers! What to replace INIT_PARAMETERS_INCORRECT with ? - page 4

 
fxsaber:

I've made a test Expert Advisor for the Optimizer's mat mode

This is what an optimization graph looks like for 8 agents in parallel in full brute force mode without interleaving passes (Rand = false)

One can clearly see here that Agents are executing tasks in batches.


In order to see dependence on parameter, let's switch to display mode

Here it is - a native parabola, top of which (unit) we will look for through GA.


In shuffle mode (Rand = true), of course, full brute force finds our vertex


 
fxsaber:

Here is the native parabola, the top of which (one) will be sought through GA.

Without GA mixing I did not find the extremum, but came close to it

Tester  Best result 0.9992924399999999 produced at generation 0. Next generation 1
Tester  Best result 0.9992924399999999 produced at generation 0. Next generation 2
Tester  Best result 0.9992924399999999 produced at generation 0. Next generation 3
Tester  Best result 0.9992924399999999 produced at generation 0. Next generation 4
Tester  Best result 0.9992924399999999 produced at generation 0. Next generation 5
Tester  Best result 0.999424 produced at generation 5. Next generation 6
Tester  Best result 0.99987456 produced at generation 6. Next generation 7
Tester  Best result 0.99987456 produced at generation 6. Next generation 8
Tester  Best result 0.99987456 produced at generation 6. Next generation 9
Tester  Best result 0.99987456 produced at generation 6. Next generation 10
Tester  Best result 0.99987456 produced at generation 6. Next generation 11
Tester  Best result 0.99987456 produced at generation 6. Next generation 12
Tester  Best result 0.99987456 produced at generation 6. Next generation 13
Tester  Best result 0.99987456 produced at generation 6. Next generation 14
Tester  Best result 0.99987456 produced at generation 6. Next generation 15
Tester  Best result 0.99987456 produced at generation 6. Next generation 16
Tester  genetic calculation is over
Tester  result cache used 973 times
Tester  genetic optimization finished on pass 1152 (of 10001)
Statistics      optimization done in 0 minutes 00 seconds
Statistics      local 179 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%)

It took 179 passes. Let me remind you, the complete set of tasks is 10001 passes. Yes, the problem is very simple, but still.


Now let's try the variant with shuffling (Rand = true).

Tester  Best result 0.99949824 produced at generation 0. Next generation 1
Tester  Best result 0.99949824 produced at generation 0. Next generation 2
Tester  Best result 0.99949824 produced at generation 0. Next generation 3
Tester  Best result 0.99949824 produced at generation 0. Next generation 4
Tester  Best result 0.99992256 produced at generation 4. Next generation 5
Tester  Best result 0.99992256 produced at generation 4. Next generation 6
Tester  Best result 0.99992256 produced at generation 4. Next generation 7
Tester  Best result 0.99992256 produced at generation 4. Next generation 8
Tester  Best result 0.99992256 produced at generation 4. Next generation 9
Tester  Best result 0.99992256 produced at generation 4. Next generation 10
Tester  Best result 0.99992256 produced at generation 4. Next generation 11
Tester  Best result 0.99992256 produced at generation 4. Next generation 12
Tester  Best result 0.99992256 produced at generation 4. Next generation 13
Tester  Best result 0.99992256 produced at generation 4. Next generation 14
Tester  Best result 0.99992256 produced at generation 4. Next generation 15
Tester  Best result 0.99992256 produced at generation 4. Next generation 16
Tester  genetic calculation is over
Tester  result cache used 970 times
Tester  genetic optimization finished on pass 1152 (of 10001)
Statistics      optimization done in 0 minutes 01 seconds
Statistics      local 182 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%)

Close again. It's clearly seen in the log, 182 passes were fully calculated, and 970 were taken from cache.


Good result, therefore, let's try to increase number of points in interval.

 
fxsaber:

This is a good result, so let's try increasing the number of points in the interval.

The number of points in the interval is one million.


Rand = false

Tester  Best result 0.9999999424 produced at generation 3. Next generation 32
Tester  genetic calculation is over
Tester  result cache used 7533 times
Tester  genetic optimization finished on pass 8704 (of 1000001)
Statistics      optimization done in 0 minutes 04 seconds
Statistics      local 1171 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%)


Rand = true (out of a million points, randomly chosen pairs were swapped 100 million times)

Tester  Best result 0.999998294364 produced at generation 10. Next generation 32
Tester  genetic calculation is over
Tester  result cache used 7452 times
Tester  genetic optimization finished on pass 8704 (of 1000001)
Statistics      optimization done in 0 minutes 04 seconds
Statistics      local 1252 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%)


The result shows that my self-assured statement

Forum about trading, automated trading systems and trading strategies testing

Not for MT Developers! What to replace INIT_PARAMETERS_INCORRECT with ?

fxsaber, 2018.07.10 16:27

Obviously if you trace a full enumeration of y = x^2. Then randomly shuffle the opimization rows and create a new set based on the shuffling. Then the GA will not find the vertex of the parabola.

Is rather false than true.


On the other hand, the choice of fitness function as a parabola was extremely short-sighted initially, as the number of points close to one is off the charts.


There is only one question for the GA, why the exact extremum was not found without mixing, since the chosen function is ideal for this?

 
fxsaber:

On the other hand, choosing a fitness function as a parabola was extremely short-sighted initially, as the number of points close to one is off the charts.

I suppose if you choose a function with a sharp peak, the GA will still break after mixing. And such a function is much closer to TC.

 
fxsaber:

I've put out a test EA for the Optimizer's math mode

Unfortunately, my knowledge is not enough to understand it all ))))

I decided to make it simpler. In general, I added to my Expert Advisor a record in the file of legitimate chains. I have found 1953 possible variants out of 117649. By the way, it took me 3 days to do the full search on interval of 3 days, only open prices and 0 trades.

Tester	optimization finished, total passes 117649 (successful 114123 passes)
Statistics      optimization done in 2 hours 36 minutes 36 seconds

I have a file now. I don't know yet how to use it for optimization. I will think about it tomorrow...

And why are there fewer successful ones? Will we have to do it again?
 
fxsaber:

I assume that if you select a function with a sharp peak, the GA will still break after mixing. And such a function is much closer to TC.

So genetics is not designed to catch peaks. Its purpose is to find stable regions, which is what TC requires. And peaks are noise, as a rule.

As for incorrect parameter combinations, they should be eliminated at population formation stage, i.e. if gene exchange or mutation results in incorrect individual, it is necessary to repeat its generation again until the correct one is obtained. Then everything will be OK.

If sifting out byINIT_PARAMETERS_INCORRECT occurs after the population is formed, thereby reducing the size of this population, then of course this violates the normal mechanism of genetic selection. Apparently MQ has this variant. And in this case the task of the topicstarter is hardly solvable (in general case), dancing with tambourines will not help.

 
Сергей Таболин:

Thanks for your opinion, but there's no discussion here about the architecture of the programme. In case you haven't noticed...

Wrong, wrong.

I agree that your problem is exactly the wrong architecture of the program. I think you can only get the maximum by full brute force, and it will be highly unstable.

 
Well, I'll put in another 5 kopeck.
Most likely full overshoots are used on every tick. This in itself leads to slowness of operation. If so, we should consider using full searches only at Expert Advisor's start/open bar and then checking for the latest data. Such optimization always reduces resource consumption.
 
Alexey Navoykov:

Genetics is not designed to catch peaks. Its purpose is to find stable regions, which is what TC requires. And peaks are noise, as a rule.

Not exactly so. Moreover, even stable areas of optimization space are still areas with dips and peaks, the FF is far from smooth.

Genetics is a versatile tool, it could be tweaked to optimise a corrupted FF but in this case it will take longer to optimise a smooth FF.

Imho, standard genetics should digest sets with half of INIT_PARAMETERS_INCORRECT without even thinking about it. Especially since these areas are usually well clustered.

but in general the best solution is to transform the input parameter space.

 
Georgiy Merts:

In vain, in vain.

I agree that your problem is in the wrong architecture of the program. I think you can only get the maximum by full brute force, and it will be very unstable.

It's not about architecture. It's about the set of functions and the order in which they are applied. In fact, originally, these functions had a single constant sequence. Then I decided that perhaps I should experiment with the order of calling them and not calling some of them. I implemented it through INIT_PARAMETERS_INCORRECT. You may see the progress. But I have a problem with genetic optimization.

Now to the point. I've created a file with legitimate chains. How to use it now? I suspect, that I should use onTester, frames... Looked at the documentation, but something doesn't add up. Can't figure out how to deal with it.

I'm thinking of reading it into an array and taking data from it...