MT5 terminal updated today and the "Optimisation" window does not show up during the test - page 4

[Texnolog]

Renat Fatkhullin:

Why didn't you say from the start that you were running genetics?

The very principle of genetic search is based on randomness and so it has no point in "continuing what you started". Its job is to run randomly.

And even then the cache is used. It's just that your probability of hitting previously calculated results is very low (you have a search area of 50 million variants). So it looks like starting from the beginning, since the starting point is randomized.

If you read the help, it says exactly the opposite. Maybe it is written incorrectly? I would like to understand.

https://www.metatrader5.com/ru/terminal/help/algotrading/optimization_types

• If the total number of optimization steps exceeds 1 000 000 in a 32-bit system or 100 000 000 in a 64-bit system, the quick optimization mode will be automatically applied.
• In genetic optimization, intermediate results are saved in the cache after calculation of each generation (file_data_platform/tester/cache/*.gen). Thus, the genetic optimization process can be interrupted at any moment. Even if the process of genetic optimization is interrupted due to external causes (for example, a power outage), the optimization will automatically continue from the last calculated generation at the next start. The genetic optimization cache is stored until the optimization settings are changed or the optimization process is complete.
• If optimization is stopped normally (with the "Stop" button), all previously calculated passes will be stored. If optimisation is resumed, the process will continue from where it was stopped.

[Renat Fatkhullin]

So it is written above that we have completely changed the way the cache works. The documentation will change over time.

Plus I've shown how in genetics cache hits dramatically improve re-run results. From 50%.

The point of a genetic tester is that it has to be run multiple times so that the randomizer can get out of local extremes. This was just shown in the screenshot above, where the fourth run found a new slide.

You always have to understand the mechanics of the process. If the solution search space is in the trillions, then the genetics have to be given a lot of tries.

[Vitaly Muzichenko]

Renat Fatkhullin:
It is written above that we have completely changed the way the cache works. We will change the documentation in time.

Plus I've shown how in genetics cache hits dramatically improve re-run results. From 50%.

The point of a genetic tester is that it has to be run multiple times so that the randomizer can get out of local extremes. This is just shown in the screenshot above, where the fourth run found a new slide.

You always have to understand the mechanics of the process. If the solution search space is in the trillions, then genetics has to be given a lot of tries.

Thanks for the hard work!

I would like to see normal statistics for each symbol in multicurrency testing, at least as a minimum.

[Texnolog]

Renat Fatkhullin:

Why don't you attach a description for reproduction?

Don't you understand that a picture is not evidence in the complete absence of a description? You are on a technical forum discussing technical issues.

I tried to build according to your conditions:

• your standard Moving Average.set
  
• your set of settings for 50 million passes
  
• MetaQuotes-Demo server, EURUSD H1, last year, Open Prices mode with genetics
  
• ran genetics

Here is the result: everything is normal and reasonable, the tester works, the genetics finds the right results

You can see how the genetics has got to the green best results zones and has consolidated there.

The Expert Advisor is a standard Moving Average.

I have carried out 4 optimizations. Log files are in appendix. This is all with genetics.

I made 2 on new build. 2 on old build.

1st run - it took 3.56 min. on new one without stopping from start to finish.

3rd run - with old one without stopping from start to finish took 2.15 min.

2nd run - the new one with one stop from the beginning to the end took 2.25+7.57= 9.82 min.

The 4th run - on the old one with one stop from start to finish took 1.55+1.17= 2.72 min.

Tell me why the new build works so badly.

[Alexey Navoykov]

Renat Fatkhullin:
Thus, it is written above that we have completely changed the way the cache works. We will modify the documentation in due course.

Plus I've shown how in genetics cache hits dramatically improve re-run results. From 50%.

The point of a genetic tester is that it has to be run multiple times so that the randomizer can get out of local extremes. This is just shown in the screenshot above, where the fourth run found a new slide.

You always have to understand the mechanics of the process. If the solution search space is in the trillions, then the genetics has to be given many tries.

If you need to run it several times, why not implement all this in the algorithm? And you can understand the mechanics of the process when you have enough information about the object. But here the user is given a black box of which he has only a general idea without knowing details (population size, stopping criteria etc.) And hence he cannot control these parameters. So if the optimizer clings to local extrema, the algorithm parameters are probably not set quite right.

[Renat Fatkhullin]

Texnolog:

Advisor is a standard Moving Average. set attached

Tell me why the new build works so badly.

Please state your question accurately.

[Renat Fatkhullin]

Alexey Navoykov:

If you need to run it several times, why not implement it all in the algorithm? And the mechanics of the process can be understood when we have enough information about the object. Here the user is provided with a black box of which he has only a general idea, not knowing details (population size, stopping criteria, etc.). And hence he cannot control these parameters. So if the optimizer clings to local extrema, the algorithm parameters are probably not set quite right.

Because this is the essence of the genetic method: in an infinite search space to output a fast solution in N generations when the increment of the target function falls below a certain level.

Key word: fast.

Genetic algorithm doesn't guarantee absolutely exact result. It's for quick and dirty estimation to get the first analytics.

How genetics should be used:

1. Set broad search limits on billions/trillions of variants
2. Do the first run.
3. Using visualization in different planes, including 3D, see where there are decent results
4. Narrowed down the search, turning some parameters off (by locking or narrowing down to a reasonable number)
5. Started a new run
6. Then we got some more results compared to the first run
7. Re-estimated with visualization
8. Proceed to step 4 until you are satisfied with the results
   

Don't turn your head off when you see 10,000 runs in the search space for trillions of variants. Miracles don't happen.

Genetics should be used thoughtfully and manually.

In general, I recommend delving deeper into the theory. Without it, I see genetics as a "push the button and that's it" attitude:

• Strategy optimization - Algorithmic trading, trading robots

• Genetic Algorithms in MetaTrader 4. Comparison with the optimizer direct brute force

• Genetic algorithms are easy!

• Genetic Algorithms - Mathematical Apparatus

• Self-Optimizing Experts: Evolutionary and Genetic Algorithms

• Guided Optimization: Annealing Techniques

• Walk-Forward Optimization in MetaTrader 5 - Your Own Hands

• Create Your Own Criteria for Optimizing Expert Advisor Parameters

• Applying Monte Carlo Method for Trading Strategy Optimization

• Random Decision Forest in Training with Reinforcement

• The optimization (testing) technique and some criteria for the Expert Advisor working parameters selection

• How to avoid optimization traps?

[Renat Fatkhullin]

Vitaly Muzichenko:

Thank you for your hard work!

I would like to see normal statistics for each symbol in the multi-currency test, at least as a minimum

We have started a big upgrade of the tester, rewriting it in parts. Now we are busy changing the scheme of working with data.

We will change a lot of things, including reporting.

[Aleksey Vyazmikin]

Renat Fatkhullin:

We are busy with a big performance upgrade of the tester and are redesigning the heavy load modes. Major improvements have been made and new acceleration methods will be implemented soon.

We decided to show the pass list window at the end of the run, so as not to waste any real resources on maintaining, re-sorting and displaying the ever-changing pass list.

There really was a huge waste of resources and slowdowns. Especially when we're talking hundreds of thousands of rows, millions and tens of millions of passes. There's no reasonable sense in looking at a bunch of preliminary data with your eyes.

We are running optimization and run tests with 100 million full passes.

It's clear that with such numbers it's out of the question to re-sort and show a table of 2-5-10-50 million values in real time. The only option is to gather everything quickly and economically, sort it finally and provide browsing at any depth.

All this is great!

However, please return the tab "Optimization" - it is impossible to work without operational analytics! Large tables are slow - let's make a filter - show the top 20 for each criterion - it is not so resource-intensive(?), but it will help to see the picture very much. And, those billions of passes - who makes them? They are units with huge capacities - you yourself are talking about the reasonableness of applying genetics, and there are no such portmanteaus there. I.e. more than 10k passes is a rarity.

I don't understand about stopping of optimization, if I have frames written to file, then after the restart a new file will be written, because I can not tell the Expert Advisor that they continued from where they stopped and do not need to create a new file for recording the results. And if that is the case, we cannot do without monitoring the current results!

If I am not so sure, you may create a graph with the current optimization results for different indicators that will help me switch the graph to the necessary one and monitor the situation in this way.

I will ask you to switch parameter "optimization" to one-pass testing after optimization. I often lost my data because of it and there were many complains on this point.

Concerning the agents in the local network - please teach them not to rub the cache and the entire environment for a longer time - between optimizations time passes to analyze data, and not immediately run for 5 minutes. Now it turns out that as soon as I start optimization again and starts distribution of whatever it is - agents start working in 10 minutes. Yes, I have a slow internet for uploading, but is it possible to give out so much? Why can't I give to each local computer (one agent on this PC) and let it divide everything between agents!

[Texnolog]

Renat Fatkhullin:

State the question precisely, please.

Why break a good old algorithm and replace it with a new one that is 3 times slower?

If you need to introduce a new approach to the genetic method, add a new item to the "optimization" tab,

Create a description and techniques for working with it.