Question for developers - using all computational cores during optimisation - page 7

 
Boris Egorov:

second day it does not count anything, all the cores in the number of 12 local and about 30 network cores are idle, I do not touch it on purpose ... I don't know what it's thinking, probably looking for the meaning of life or a cure for coronovirus :-)

I think we should abandon the optimizer because of its inoperability and sluggishness

and the recent decisions made by MT such as to limit only physical cores, persistently and stupidly distribute a bunch of tasks only to certain cores and not to each core - one task - indicates a total lack of understanding by developers of high-performance calculations

Why don't you show a full screenshot of the agent list rather than part of it?

Tester logs.

Resource Monitor.

It looks like you have overloaded not only your computer (on the subject of logical and physical cores), but also the farm

 
Slava:

Why don't you show a full screenshot of the agent list rather than part of it?

Tester logs.

Resource monitor.

Looks like you overloaded not only your computer (speaking of logical and physical cores) but also the farm

no problem

Only it's the same... available and idle.

I don't see anything in the logs to complain about

I did not get it wrong ... read the post above

 

By the way, there is a separate sad story about logs

Many times the developers were asked to turn them off or limit their size - and the result is zero.

as a result, logs eat up all disk space .... with all that this implies.

I delete logs episadically ... I don't need log files as a class and if there are any errors, they are immediately visible in the log window

 

here's just a little something from yesterday


 
confirmed memory overload .... although strange, swap is still around, again I think the developers need to take this into account
 
Boris Egorov:
confirmed information about memory overload .... Although strange, no one cancelled swap, again, I think developers need to take it into account

Maybe you showed it, I didn't look - just in case, I will say that 1Gb of memory is recommended per core (with more cores you can try 0.5Gb). And I recommend pagefile to double memory size, not half, as usually recommended. I've proven this in practice.

 

>1GB of memory per core is recommended

will take into account

 

It is possible either by using a custom task manager for agents (I did that), or by changing the algorithm of the regular optimizer (the fixes are not significant, by the way), but the developers are not ready to listen to my suggestions, unfortunately.

I implemented custom task manager for agents for my italian customer, he uses rented powerful servers with several hundreds of agents and he constantly suffered from incomplete loading of servers, because agents were idle. now all agents are fully loaded regardless of which agent completes the task first (regardless of task completion time on each agent).

 
Andrey Dik:

I implemented a custom task manager for agents for my italian customer, he uses rented powerful servers with several hundred agents and he constantly suffered from under-utilization of servers, because agents were idle. now all agents are fully loaded no matter which agent completes the task first (no matter how long it takes each agent to complete the task).

It's not nice to brag )

What is the manager based on? How are the tasks distributed?

 
Andrey Khatimlianskii:

Bragging is not good )

What is the manager based on? How are tasks allocated?

bragging? - Nah, I can't get anywhere near the quality and functionality of MT5 if I want to write an analytical trading platform for myself, it's easier to stimulate developers with ideas ...))

In an optimized EA, an additional input parameter counter is added, which is searched by MT5 optimizer in full search mode, you can set up to 100 mio steps, so the optimizer doesn't switch to genetics. The manager hangs in the form of an EA on the chart in the terminal and writes tasks in the agents folder, and the optimized EA in the optimizer reads tasks and writes results on completion of the pass. In the mode of a complete search the optimizer MT5 loads agents very evenly, and the manager's task is to feed the agents with input parameters.

The problem was only to write a ha in the manager that could work with individuals in the population in streaming mode without the need to fill the population completely before starting a new epoch. i solved that. in essence i abolished the concept of "epoch", the individuals are born and die on their own. i suggested the same mechanism to developers, and such a laifhack can be used not only with ha. but also with other aDs, ant, particle swarm and others, any.