Evaluating CPU cores for optimisation

 

I suggest collecting statistics on the performance of various processors to estimate their effectiveness for the Strategy Tester in optimization mode.

To be more objective, I suggest using an Expert Advisor that runs in the"Mathematical Computing" mode which allows you to minimize the use of your computer's hard drive and memory. If possible, specify the name of the processor, motherboard and frequency of RAM.

The Expert Advisor has only one parameter which should be optimized (in fact, it is nothing and the data are identical for all agents - to be objective), specify the number of passes equal to the number of threads and select agents by the number of processor cores (no hypertraining!).

I hope that the collected data will help people in choosing a hardware for optimization which is reasonable to take not of the first order, at least it has been so lately, but something can change.

The Expert Advisor itself is a cut-out of a real EA I use in a similar task to select leaves from a decision tree, i.e. it is a real MoD task and if there are any ideas how to speed up the code I would be very interested to hear them. There are only 1000 leaves in the code, in reality there are already more than 70,000, so they are put into a function for an acceptable wait time for the outcome of compilation.

2019.08.09 23:04:26.397 Terminal        Windows 7 Service Pack 1 (build 7601) x64, IE 11, Intel Core i5  M 450 @ 2.40 GHz, Memory: 2045 / 3766 Mb, Disk: 53 / 148 Gb, GMT+3
In the Strategy Tester log via the context menu, tick the "Full Optimisation Log" box.
2019.08.09 22:41:25.630 Core 2  pass 2 returned result 1001000.00 in 0:04:50.391
2019.08.09 22:41:26.642 Core 1  pass 0 returned result 1001000.00 in 0:04:51.365
2019.08.09 22:46:09.036 Core 2  pass 3 returned result 1001000.00 in 0:04:43.441
2019.08.09 22:46:10.759 Core 1  pass 1 returned result 1001000.00 in 0:04:44.152
2019.08.09 22:46:10.759 Tester  optimization finished, total passes 4
2019.08.09 22:46:10.769 Statistics      optimization done in 9 minutes 36 seconds
2019.08.09 22:46:10.769 Statistics      shortest pass 0:04:43.441, longest pass 0:04:51.365, average pass 0:04:47.337

Added: In this thread it turned out that the file size in some (this is to be found out) processors affects the performance of the system as a whole, although the code structurally doesn't change much. Therefore I ask everyone to test two testers.

Example of tester settings - symbol, period and timeframe are not important - the other settings are relevant. The screenshot shows another EA, but the settings are correct


So far, we have the following rating where the average time in seconds has been used - the two last two columns and the last column shows number of passes of the processor in 1 hour.

The table is filtered by the last column, as the heaviest EA option in terms of computational resource consumption.



Распределенные вычисления в сети MQL5 Cloud Network
Распределенные вычисления в сети MQL5 Cloud Network
  • cloud.mql5.com
Большую часть времени современные компьютеры простаивают и не используют всех возможностей процессора. Мы предлагаем задействовать их с пользой. Вы можете сдавать мощности вашего компьютера другим участникам нашей сети для выполнения разнообразных...
Files:
 
Aleksey Vyazmikin:

I suggest collecting statistics on the performance of different processors in order to estimate their efficiency for the strategy tester to work in optimisation mode.

Good initiative, I support it.

I get a warning when compiling:

implicit conversion from 'number' to 'string'   Tree_Brut_TestPL.mq5    2567    16

Is it supposed to be like this?

String:

   if(FrameAdd(Test_P,1,0,stat_values)==false)
 
Aleksey Vyazmikin:

I suggest collecting statistics on the performance of various processors to estimate their effectiveness for the Strategy Tester in optimization mode.

To be more objective, I suggest using an Expert Advisor that runs in the"Mathematical Computing" mode which allows you to minimize the use of your computer's hard drive and memory. If possible, specify the name of the processor, motherboard and frequency of RAM.

The Expert Advisor has only one parameter which should be optimized (in fact it is nothing and the data are identical for all agents - to be objective), specify the number of passes equal to the number of threads and select agents by the number of processor cores (no hypertruding!).

I hope that the collected data will help people in choosing a hardware for optimization which is reasonable to take not of the first order, at least it has been that way lately, but some things may have changed.

The Expert Advisor itself is a cut-out of a real EA I use in a similar task to select leaves from a decision tree, i.e. it is a real MoD task and if there are any ideas how to speed up the code I would be very interested to hear them. There are only 1000 leaves in the code, in reality there are already more than 70,000, so they are put into a function for an acceptable compile-time expectation.

Over what period should you optimise and on what time frame? Is the tool important?
 
Serhii Shevchuk:

Good initiative, I support it.

It gives a warning when compiling:

Is this the way it should be?

Line:

Corrected and relaunched!

Test it.

 
Maxim Romanov:
Over what period to optimise and on what timeframe? Is the tool important?

InMathematical Calculation mode it doesn't matter, no ticks are generated there!

 
What about it? Is the stump of the late 2000s no longer taking off? :D
 
Artem Prischepa:
What's out there? Is the late 2000's stump no longer taking off? :D

Try out your processor and post your results - see if it takes off or just rolls!

 

Does an old horse spoil the furrow?

hardware

Results:

2019.08.09 23:22:07.472 Tester  set "Custom max" as optimization criterion for mathematical calculations
2019.08.09 23:22:07.540 Experts optimization frame expert Tree_Brut_TestPL (EURUSD.m,H1) processing started
2019.08.09 23:22:07.592 Tester  cache file 'tester\cache\Tree_Brut_TestPL.30.DFF2DB61B8A3751199D61AD0A0F226D3.opt' deleted
2019.08.09 23:22:07.623 Tester  Experts\Tree_Brut_TestPL.ex5 math calculations test means no history and no symbol info for EURUSD.m
2019.08.09 23:22:07.623 Tester  complete optimization started
2019.08.09 23:22:07.648 Core 1  agent process started on 127.0.0.1:3008
2019.08.09 23:22:07.650 Core 2  agent process started on 127.0.0.1:3009
2019.08.09 23:22:07.652 Core 3  agent process started on 127.0.0.1:3010
2019.08.09 23:22:07.654 Core 4  agent process started on 127.0.0.1:3011
2019.08.09 23:22:07.657 Core 5  agent process started on 127.0.0.1:3012
2019.08.09 23:22:07.659 Core 6  agent process started on 127.0.0.1:3013
2019.08.09 23:22:07.662 Core 7  agent process started on 127.0.0.1:3014
2019.08.09 23:22:07.664 Core 8  agent process started on 127.0.0.1:3015
2019.08.09 23:22:07.972 Core 1  connecting to 127.0.0.1:3008
2019.08.09 23:22:07.973 Core 1  connected
2019.08.09 23:22:07.983 Core 1  authorized (agent build 2097)
2019.08.09 23:22:07.997 Core 1  common synchronization completed
2019.08.09 23:22:08.035 Core 3  connecting to 127.0.0.1:3010
2019.08.09 23:22:08.036 Core 3  connected
2019.08.09 23:22:08.046 Core 3  authorized (agent build 2097)
2019.08.09 23:22:08.056 Core 2  connecting to 127.0.0.1:3009
2019.08.09 23:22:08.057 Core 2  connected
2019.08.09 23:22:08.059 Core 3  common synchronization completed
2019.08.09 23:22:08.067 Core 5  connecting to 127.0.0.1:3012
2019.08.09 23:22:08.068 Core 5  connected
2019.08.09 23:22:08.068 Core 2  authorized (agent build 2097)
2019.08.09 23:22:08.082 Core 2  common synchronization completed
2019.08.09 23:22:08.083 Core 5  authorized (agent build 2097)
2019.08.09 23:22:08.103 Core 7  connecting to 127.0.0.1:3014
2019.08.09 23:22:08.104 Core 7  connected
2019.08.09 23:22:08.119 Core 7  authorized (agent build 2097)
2019.08.09 23:22:08.125 Core 5  common synchronization completed
2019.08.09 23:22:08.140 Core 7  common synchronization completed
2019.08.09 23:22:08.150 Core 4  connecting to 127.0.0.1:3011
2019.08.09 23:22:08.151 Core 4  connected
2019.08.09 23:22:08.151 Core 8  connecting to 127.0.0.1:3015
2019.08.09 23:22:08.151 Core 8  connected
2019.08.09 23:22:08.162 Core 8  authorized (agent build 2097)
2019.08.09 23:22:08.164 Core 4  authorized (agent build 2097)
2019.08.09 23:22:08.184 Core 6  connecting to 127.0.0.1:3013
2019.08.09 23:22:08.185 Core 6  connected
2019.08.09 23:22:08.208 Core 6  authorized (agent build 2097)
2019.08.09 23:22:08.228 Core 4  common synchronization completed
2019.08.09 23:22:08.240 Core 6  common synchronization completed
2019.08.09 23:22:08.250 Core 8  common synchronization completed
2019.08.09 23:24:45.931 Tester  optimization finished, total passes 8
2019.08.09 23:24:45.941 Statistics      optimization done in 2 minutes 38 seconds
2019.08.09 23:24:45.941 Statistics      shortest pass 0:02:35.945, longest pass 0:02:37.669, average pass 0:02:37.006
2019.08.09 23:24:45.941 Statistics      8000 frames (3.14 Mb total, 412 bytes per frame) received
2019.08.09 23:24:45.941 Statistics      local 8 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%)
2019.08.09 23:24:45.988 Tester  8 new records saved to cache file 'tester\cache\Tree_Brut_TestPL.30.DFF2DB61B8A3751199D61AD0A0F226D3.opt'

Terminal version:

2019.08.09 23:20:35.776 Terminal        MetaTrader 5 x64 build 2085 started (MetaQuotes Software Corp.)
2019.08.09 23:20:35.778 Terminal        Windows 10 (build 17763) x64, IE 11, UAC, AMD FX-8300 Eight-Core Processor , Memory: 17102 / 24574 Mb, Disk: 2407 / 2441 Gb, GMT+2
 
Serhii Shevchuk:

Does an old horse spoil the furrow?

Results:

Terminal version:

It seems that you have 8 agents involved, and you need 4, because FPU in FX have only 4 and 8 ALUs, and it is FPU which is loaded during optimization. Why so - is not clear - question to developers, because the code is built largely on the comparison, but apparently the consumption of floating point calculation prevails or times slower, so it turns out that it is necessary to set the number of coprocessors, then 1 pass will be faster and the average time will remain the same.

 
Aleksey Vyazmikin:

It seems that you have 8 agents involved, and you need 4, because FPU in FX is only 4 and 8 APUs, and it is FPU that is loaded during optimization. Why it is so - is not clear - question to developers, because code is mostly based on comparison, but apparently consumption of floating point calculation prevails or is times slower, so it turns out that it is necessary to put exactly number of coprocessors, then 1 pass will be faster and average time will approximately remain the same.

Please clarify. I turn off 4 agents and set the parameter to be optimized so that I get 4 passes of optimization or 8 passes?
 
Serhii Shevchuk:
Please clarify. Do I disable 4 agents and set the parameter to be optimized so that I get 4 passes of optimization or 8 passes?

8 passes 4 agents, kind of analogous to hypertrading - according to the number of threads.

Turn on"Full optimisation log" to see the time of each pass.