MT5 and speed in action - page 69

 
Andrei Trukhanovich:

how do you envisage this - parallel processing in one thread?

An event loop.
And in general it's a developer's problem why everything is executed in one thread.

It turns out that the Market Overview is processed asynchronously, and the handlers in the user programs, synchronously.
It's just chicardous, there are no words.

 

Thanks for the feedback on the stats! The OnTick/OnBook lags seem to be there for everyone. Sleep(1) is either 1 ms or 15 ms. It's not clear why it depends.

 
fxsaber:

Everyone seems to have OnTick/OnBook lags.

I think you know that OnTimer() cannot be called more often than 10-16 milliseconds. It is logical to assume that other OnXXX functions are not called more often either. Maybe that's why your lags are occurring ?

 
pivomoe:

I think you know that OnTimer() cannot be called more often than 10-16 milliseconds. It is logical to assume that other OnXXX functions are not called more often either. Maybe that's the reason for your lags?

Not logically, the handlers aren't tied to GetTickCount timer frequency/resolution. Events are triggered instantly the moment they happen.

OnTimer isn't bound to 16ms error either. It's easy to get a 1ms timer in the vast majority of cases, unless the computer is dead and 100% loaded.


The problem with almost all of fxsaber's tests is that he tries to measure the outliers of single calls instead of averaging the set and doesn't want to understand the reality of the thread dispatcher.

He has:

  • at least 1500-2000 threads on 4/8 cores
  • the poor fellow thread manager distributes threads over an insanely small number of actors -people don't realize that their task has hundreds of competitors
  • occasionally priority threads wake up which eat quantum of time more than others for a short time
  • Any thread can be moved away from the trough for a significant time at any moment - that is milliseconds on a trivial i++ (how many times must I say it?).

Methods of fighting to get closer to reltime:

  • more cores to destroy the thread manager
  • definitely new CPUs, with modern caches, good clock speed and increased IPC
  • more fast memory and nvme disks, which dramatically removes the impact of any third-party appetites
  • correct drivers and bios, so that the commonplace network interface does not silently sabotage interrupts (especially monstrous in virtual machines, where ISP admins are not only unaware of the problem, but they are generally unfamiliar with latency, SR-IOV and deblocking/unlocking bottlenecks)
  • a mid-sized discrete graphics card capable of relieving any 2D loads of operating system and program interfaces rendering (always a pain on servers and virtual desktops)
  • decrease in number of unused processes/programs
  • decreasing amount of unnecessary peripheral hardware and drivers

Consequently, on a regular VPS the terminals (as well as any other programs) will always suffer from unexpected delays. The more cheap/over-sold VPS, the more problems.

Ask yourself on your VPS, is SR-IOV enabled and are there any latest (manually installed only) special network drivers for it? In 99% of cases, no, as it breaks the migration of virtualisations across the hardware zoo. And without it additional delays are guaranteed simply because of double (on host and virtual) network packet transmission/processing and increased number of interrupts.

Our VPS service is much less prone to this, both in terms of architecture and lightweight terminals without GUI.

 
Renat Fatkhullin:

Not logical, handlers are not tied to GetTickCount timer frequency/resolution. Events are triggered instantly at the time of occurrence.

OnTimer is not bound to 16ms error either. It's easy to get a 1ms timer in the vast majority of cases, unless the computer is dead and 100% loaded.


The problem with almost all of fxsaber's tests is that he tries to measure the outliers of single calls instead of averaging the set and doesn't want to understand the reality of the thread dispatcher.

He has:

  • at least 1500-2000 threads on 4/8 cores
  • the poor fellow thread manager distributes threads over an insanely small number of actors -people don't realize that their task has hundreds of competitors
  • occasionally priority threads wake up which eat quantum of time more than others for a short time
  • Any thread can be moved away from the trough for a significant time at any moment - that is milliseconds on a trivial i++ (how many times must I say it?).

Methods of fighting to get closer to reltime:

  • more cores to destroy the thread manager
  • definitely new CPUs, with modern caches, good clock speed and increased IPC
  • more fast memory and nvme disks, which dramatically removes the impact of any third-party appetites
  • correct drivers and bios, so that a trivial network interface does not quietly sabotage interrupts (especially monstrous in virtual machines, where ISP admins are not only unaware of the problem, but they are generally unfamiliar with latency, SR-IOV and debottlenecking)
  • a mediocre discrete graphics card capable of relieving any 2D loads of operating system and program interfaces rendering (always a pain on servers and virtual desktops)
  • decrease in number of unused processes/programs
  • decreasing amount of unnecessary peripherals and their drivers

Consequently, on a regular VPS the terminals (as well as any other programs) will always suffer from unexpected delays. The more cheap/over-sold VPS, the more problems.

Ask yourself on your VPS, is SR-IOV enabled and are there any latest (manually installed only) special network drivers for it? In 99% of cases, no, as it breaks the migration of virtualisations across the hardware zoo. And without it additional delays are guaranteed simply because of double (on host and virtual) network packet transmission/processing and increased number of interrupts.

Our VPS service is subject to it by orders of magnitude less, both in terms of architecture and lightweight terminals without GUI.

And now imagine how many times slower performance of user programs would be on such an optimized machine, if handlers in programs would be executed asynchronously.
I do not understand the sense of super upgrade and super optimization of machine hardware, if handlers in user's program are a priori executed synchronously.
For the terminal itself and its inner workings, perhaps, the optimization described above is useful. For user fighting programs, I doubt it.
Because consecutive execution of handlers in user program reduces all that potential of any super-optimized machine.
The problem is not in the hardware, but in the architecture of the internal work of the terminal.

 
Roman:

And now imagine how many times faster running of user's programs will be on such an optimized machine, if handlers in programs will be executed asynchronously.
I do not understand the meaning of super upgrade and super optimization of machine hardware, if handlers in user's program are a priori executed synchronously.
For the terminal itself and its inner workings, perhaps, the optimization described above is useful. For user fighting programs, I doubt it.
Because consecutive execution of handlers in user program reduces all that potential of any super-optimized machine.
The problem is not in the hardware, but in the architecture of internal operation of the terminal.

There will be no acceleration. Please submit your calculations, at least in approximate figures, proving a multiple acceleration.

A race for resources? Uncontrolled creation of new threads? Conflicts over nothing?

Do you want unexplained slowdowns?

In the event-based model, all events have always gone in formation, one at a time. Chewed up - chewed up.

 
Renat Fatkhullin:

Our VPS service is much less prone to this, both in terms of architecture and lightweight terminals without a GUI.

If there are lags on your VPS service, will you take it seriously?

Who uses VPS from MQ, please share the results of the following programs there.

  1. Expert Advisor, which catches lags OnTick/OnBook.
  2. Expert Advis or that catches ticks with old time.
  3. A script that measures average execution time of Sleep(1).
 
How can I get Sleep(1) to run faster?
 
fxsaber:

If there are lags on your VPS service - will you take it seriously?

I wonder how many times I have to tell you that with so many (thousands) threads per limited number of cores it's insane to talk about stability of time quantum allocation per thread at all?


You're always guaranteed to have failures on random single samples of any instruction, including the simplest assembler ones like inc eax. This is architecturally and due to physical limitations of "honestly allocating time quanta of thousands of threads to a small number of cores".

Stop being dumb and keep catching single bursts per million requests.

 
Renat Fatkhullin:

Stop being stupid and keep catching single outliers per million queries.

I see what's going on with single spikes, thanks for the detailed explanation. At the moment we are not discussing SymbolInfoTick, but lags of another kind, which occurs on almost every tick.