MT5 and speed in action - page 45

 
Andrey Khatimlianskii:

No, I've got virtualisers spinning on CentOS. But I'm not competent to continue this dialogue.

It's double virtualisation anyway.

CentOS -> VirtualBox -> Windows 7


Also cutting down to 2 cores when the CPU is at 8, drastically changes the behaviour and allocated resources of the threadsheduler.

These 2 cores will have to be allocated to the inevitable 1000 threads even with truncated Windows 7. So the terminal is guaranteed to have increased latency.

 
Renat Fatkhullin:

Well you are a master of racing stress tests without controlling for correlation and reasonableness.

Of course microsecond metering requires resources to be able to measure gaps of 1000 times less than a millisecond.

If you occasionally need to measure intervals super accurately, then use microseconds. And it will cost you 0 microseconds.

How do you use microseconds when they're what's slowing you down?! Here's a total of 20 calls. A quarter of a millisecond on such a ridiculous number of calls for various practical tasks.

2020.10.06 00:50:44.491 Alert: Time[Test6.mq5 15: for(inti=0;i<20;i++)GetMicrosecondCount();] = 254 mсs.
2020.10.06 00:50:44.491 Alert: Time[Test6.mq5 20: for(inti=0;i<20;i++)GetTickCount();] = 19 mсs.

On a stifled VPS, overclocking the system timer via timeBeginPeriod is fraught with danger. You'll simply increase the CPU overhead:

Otherwise, you would have long ago made GetTickCount/GetTickCount64 accurate in the OS and rejoiced in the free accuracy. But no, you will have to pay for accuracy of this timer.


GetTickCount is in no way inferior in speed. Switched to using it on a slow VPS instead of GetMicrosecondsCount. The load has dropped from 50% to 2% on a real trade.

2020.10.06 00:50:44.491 Alert: Time[Test6.mq5 26: for(inti=0;i<20;i++)winmm::timeGetTime();] = 13 mсs.
 
fxsaber:

How do you use microseconds when it's the ones that are slowing down?! Here are just 20 calls. A quarter of a millisecond on such a ridiculous number of calls for various practical tasks.

And I have these 20 calls:

   ulong ticks=GetMicrosecondCount();
   for(int i=0; i<20; i++)
      GetMicrosecondCount();
   Print("GetMicrosecondCount: ",GetMicrosecondCount()-ticks);

   ticks=GetMicrosecondCount();
   for(int i=0; i<20; i++)
      GetTickCount();
   Print("GetTickCount: ",GetMicrosecondCount()-ticks);


2020.10.06 01:04:44.068 5555 (CAT.NYSE,M5)      GetMicrosecondCount: 1
2020.10.06 01:04:44.068 5555 (CAT.NYSE,M5)      GetTickCount: 0


GetTickCount is in no way inferior in speed. Switched to using it on a slow VPS instead of GetMicrosecondsCount. Load has dropped from 50% to 2% on a real trade.

I don't believe that replacing GetMicrosecondsCount -> GetTickCount will give in a real program. Both theoretically and practically there is no evidence.

In a stress test on these two functions, you can easily draw such a result. Are you concluding yourself that 48% of CPU load was measured by microseconds? And this is not a stress test? Of course it is.


About the multimedia timer acceleration - that's stress tests again without looking at overall performance degradation. Overclocking the task shuffler increases the system overhead of the OS.

 
Renat Fatkhullin:

Still dual virtualisation.

CentOS -> VirtualBox -> Windows 7


Also cutting down to 2 cores when the CPU is at 8, drastically changes the behaviour and allocated resources of the threadsheduler.

These 2 cores will have to be allocated to the imminent 1000 threads even with the truncated Windows 7. So the terminal is guaranteed to have increased latency.

I've come to that conclusion too, thanks for the confirmation.VirtualBox is evil.
And especially beware of vps, on such a deployment, there are plenty of them.
Only pure operating system on hardware, and preferably Linux.
Though through wine, same virtualization, but GUI of terminal just flies without a single lag.
And GetMicrosecondsCount rolls around without any lag.

 
Renat Fatkhullin:

And I have these 20 calls:

Well, that's zero microseconds for me too! Only on my home machine.

I don't believe replacing GetMicrosecondsCount -> GetTickCount will do in a real program. Both theoretically and practically there is no proof.

In the stress test, you may easily draw such errors using these two functions. Are you concluding yourself that 48% of CPU load was measured in microseconds? And this is not a stress test? Of course it is.

This thread prompted me to write a Benchmark library so that I could simply insert a runtime check into the source. And it took advantage of that quite extensively, revealing and eliminating a lot of bad stuff.

So, Expert Advisor written with such modification (more precisely 20 EAs in parallel) affect home computer by 1.5%. But the VPS is 50+%. When I started digging around, I saw that the microsecond timer is slowing down. Correspondingly, where no alerts were generated on home machines, the VPS fails.


But even that wasn't enough. Thanks to this branch, a snapshot mechanism was developed, the basis of which is this.

  ulong Snapshot( const uint &RefreshTime, const MAGIC_TYPE &Magic, bool HistoryInit = false )
  {
    if (SNAPSHOT::SnapshotLifeTime() < RefreshTime)
      return(0);
// ....

  ulong SnapshotLifeTime( void ) const
  {
    static const bool IsTester = ::MQLInfoInteger(MQL_TESTER);

    return(IsTester ? ULONG_MAX : (::GetMicrosecondCount() - this.TimeData)); // Обязуем любой вызов снепшота в Тестере делать полноценным.
  }

This is the basis of all snapshots: if less than the specified time has passed since the last snapshot, we do nothing. It is this approach that allows you to significantly save on resources.

Of course, the refresh time is short - by default it is one millisecond. That is why a microsecond timer is used.


The slowness of this timer caused the snapshot mechanism to collapse as it took a full-fledged snapshot an order/twice as often as on a full-fledged machine.


Such are the pies of the microsecond timer brakes. But when I switched to millisecond timer (instead of 16ms), everything started flying, even on slow VPS.

About the multimedia timer acceleration - it's stress tests again without regard to overall performance degradation. Overclocking the taskmaster increases the system overhead of the OS.

Who cares about these theories when in practice the gains are colossal. Perhaps, it does affect some games. But on VPS it was a saving straw.

 
fxsaber:

Well, it's zero microseconds for me too! Only on my home machine.

This thread prompted me to write a Benchmark library so that I could just insert a runtime check into the source. And it took advantage of this quite extensively, revealing and eliminating a lot of bad stuff.

So, Expert Advisor written with such modification (more precisely 20 EAs in parallel) affect home computer by 1.5%. But the VPS is 50+%. When I started digging around, I saw that the microsecond timer is slowing down. Correspondingly, where no alerts were generated on home machines, the VPS fails.


But even that wasn't enough. Thanks to this branch, a snapshot mechanism was developed, the basis of which is this.

This is the basis of all snapshots: if less than the specified time has passed since the last snapshot, we do nothing. It is this approach that allows you to significantly save on resources.

Of course, the refresh time is short - by default it is one millisecond. That is why a microsecond timer is used.


So, because of slowness of this timer the snapshot mechanism crashed because the full-fledged snapshot was taken much more often than on a full-fledged machine.


That's the kind of pie from the microsecond timer brakes. But when I switched to millisecond timer (instead of 16ms), everything started to fly, even on slow VPS.

I don't care about these theories, as long as the practical benefits are enormous. Maybe it does affect some games. But on VPS it was a saving straw.

See how beautifully it was said:

Switched to using it on a slow VPS instead of GetMicrosecondsCount. Load dropped from 50% to 2% on real trade.

Conclusion was inevitable "all because of microseconds metering brakes, that's what speeding up is".

And suddenly it turns out "I myself made calculations an order of magnitude larger because of a logical error". And GetMicrosecondsCount was only a trigger for this error.

Rework to GetTickCount is a fix/crack for this error and the code of the fix was not shown. Because it's not just replacement of GetMicrosecondsCount -> GetTickCount?

Why couldn't it have been said right away?


The logic seems to have been accelerated by the apparent bootstrapping of accounting (jumping from microseconds to milliseconds) and the multiple reduction of snapshot creation.
 
I'm still thinking whether to snapshot SymbolInfoTick to avoid it.
2020.10.05 12:52:35.963         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 599 mсs.
2020.10.05 12:52:45.904         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 245 mсs.
2020.10.05 12:52:45.904         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 470 mсs.
2020.10.05 12:52:45.904         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 722 mсs.
2020.10.05 12:52:45.904         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 901 mсs.
2020.10.05 12:52:45.905         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1726 mсs.
2020.10.05 12:53:00.123         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 864 mсs.
2020.10.05 12:53:03.218         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 112 mсs.
2020.10.05 12:53:04.493         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 2134 mсs.
2020.10.05 12:53:10.013         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1826 mсs.
2020.10.05 12:53:13.119         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 114 mсs.
2020.10.05 12:53:18.008         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 116 mсs.
2020.10.05 12:53:20.010         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1095 mсs.
2020.10.05 12:55:55.033         Alert: Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 359 mсs.

This is on a machine which grinds 20 EAs with 1.5% CPU. Which LatencyMon shows that everything is fine.

Something in the MT5 architecture is giving these lags to all running EAs at the same time. And none for me.

 
Renat Fatkhullin:

The conversion to GetTickCount is a fix/crack for this bug, and the fix code was not shown. Because it's not just replacement of GetMicrosecondsCount -> GetTickCount?

Why couldn't it have been said right away?

Discussions with me, though, are distinguished by the presence of what is shown. Nothing is hidden, on the contrary it is openly published.

There is certainly a stress test. There was no other way to show it visually.


Imagine a function in the form of a matryoshka doll. A measurement is taken of the outer matryoshka. At the same time, measurements are taken of the internal nesting dolls as well. As a result, due to microsecond braking, the external nested dolls show wild figures on their execution times, which causes a flurry of one-type Alerts. Obviously, 10 microseconds for one call to GetMicrosecondsCount is terribly expensive. So I rushed the alerts.


The free rough millisecond timer started giving out either 0µs, 1000µs or 2000µs. This greatly reduced the number of Alerts and reduced the braking on timer function calls.


Judging by the logic, the acceleration was obtained by explicitly bootstrapping the accounting (jumping from microseconds to milliseconds) and reducing the creation of snapshots by a multiple.


And with snapshots, it's great. The loadout, compared to the home machine (microseconds there), is not great. But if you compare it with what was on VPS, it's like heaven and earth.


SZZ we are now talking about the feasibility of having in MQL millisecond timer, which, unfortunately, does not exist. You can't snapshot on VPS without it.

 
fxsaber:
I'm still considering the SymbolInfoTick snapshot to avoid this kind of stuff.

This is on a machine which grinds 20 EAs with 1.5% CPU. Which LatencyMon shows that everything is fine.

Something in the MT5 architecture is giving these lags to all running EAs at the same time. And none for me.

Here is my code and stable response time: no hundreds or thousands of microseconds on 20 charts in parallel

   MqlTick Tick;
   ulong   ticks=GetMicrosecondCount();
   
   SymbolInfoTick(_Symbol,Tick);
   Print("SymbolInfoTick: ",GetMicrosecondCount()-ticks);


2020.10.06 02:14:18.234	5555 (CADJPY,H1)	SymbolInfoTick: 2
2020.10.06 02:14:18.765	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:19.063	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:19.180	5555 (CADJPY,H1)	SymbolInfoTick: 2
2020.10.06 02:14:19.180	5555 (EURCAD,H1)	SymbolInfoTick: 1
2020.10.06 02:14:19.245	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:19.523	5555 (CHFJPY,H1)	SymbolInfoTick: 1
2020.10.06 02:14:19.659	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.037	5555 (CADCHF,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.037	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.137	5555 (EURMXN,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.138	5555 (EURNOK,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.226	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.227	5555 (CHFJPY,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.525	5555 (AUDNZD,H1)	SymbolInfoTick: 1
2020.10.06 02:14:20.645	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:20.919	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:21.123	5555 (EURNZD,H1)	SymbolInfoTick: 1
2020.10.06 02:14:21.129	5555 (EURAUD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:21.234	5555 (EURNOK,H1)	SymbolInfoTick: 1
2020.10.06 02:14:21.441	5555 (EURAUD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:22.299	5555 (EURNZD,H1)	SymbolInfoTick: 2
2020.10.06 02:14:22.383	5555 (AUDNZD,H1)	SymbolInfoTick: 2


How many cores do you have and what kind of processor? i7-2600?

Another hidden stress test with millions of requests in parallel?


Be more transparent. Just because you posted a couple of simple _B calls is not proof of your other claims. You abruptly forget about the code and the actual description of the conditions as soon as you make outlandish claims.

You don't need to imagine anything in your mind - tell and show what you actually call and test. Not a torn out result of "ran an unknown stress test and waiting for an alert to show the world", but exactly the full code of the test.

There are also questions about the measurement library itself. There is a lot of unnecessary stuff, including overhead.

 
fxsaber:

We are now talking about the usefulness of having a millisecond timer in MQL, which, unfortunately, does not exist. You can't snapshot on UPU without it.

There is a millisecond timer long ago: EventSetMillisecondTimer()

Документация по MQL5: Работа с событиями / EventSetMillisecondTimer
Документация по MQL5: Работа с событиями / EventSetMillisecondTimer
  • www.mql5.com
Указывает клиентскому терминалу, что для данного эксперта или индикатора необходимо генерировать события таймера с периодичностью менее одной секунды. нужно получать события таймера чаще, чем один раз в секунду. Если вам достаточно обычного таймера с периодом более 1 секунды, то используйте EventSetTimer(). В тестере стратегий используется...