MT5 and speed in action - page 35

[Renat Fatkhullin]

fxsaber:

Tired of debugging the snapshots. Finally made it perfect. One advisor, nothing. Two - perfect. 20 - disaster: CPU is under 100%. HistorySelect lags for many milliseconds.

It seems that MT5 is not intended for operating many Expert Advisors at the same time.

Are you writing a stress test or a usual Expert Advisor?

Most likely exactly a multi-threaded stress test in a single base. So I will repeat:

Forum on trading, automated trading systems and trading strategy testing

MT5 and Speed in Action

Renat Fatkhullin, 2020.09.16 12:47

If I understand correctly, there is not an EA there, but a stress tester on each symbol. This changes the case completely. And it shows the hiding of initial conditions.

That is, on 8(4+HT) CPU 16 threads (+N worker terminal threads in parallel) non-stop and without delay break into one synchronized symbol database object. Read/Write locks are mixed up because there is constant tick writing.

Usually in such a profile, depending on steepness of the processor and its mastering of threads, each thread can spend from 60% to 80% of time waiting.

And this is regardless of the type of task.

If you are actually having a non-stop battle for one resource in 20 threads, there are several options:

1. logically decouple access and do not engage in stress testing
2. go to your own cache (variant of point 1)
   
3. Upgrade your hardware (don't be fooled by the fact that i7 2600k isn't bad - it's bad)
   

Read the box carefully. If N threads hit a single sync object, the empty wait will be at 60-80%.

And the limit of multi-threaded efficiency will be somewhere around 8-12 threads. As the number of threads increases, the sampling rate will drop. On 2600k the efficiency limit is even lower.

[fxsaber]

Renat Fatkhullin:

Are you writing a stress test or an ordinary expert's work?

Ordinary

Rather, it is a multi-threaded stress test in a single base. So I'll repeat:

If in fact you are doing a non-stop battle for a single resource in 20 threads, there are several options:

1. logically decouple the access and do not deal with the stress test
2. go to your own cache (variant of point 1)
   
3. Upgrade your hardware (don't be fooled by the fact that i7 2600k isn't bad - it's bad)
   

Read the box carefully. If N threads hit a single sync object, the empty wait will be at 60-80%.

And the limit of multi-threaded efficiency will be somewhere around 8-12 threads. As the number of threads increases, the sampling rate will drop. On 2600k the efficiency limit is even lower.

Fully caching history. But even this requires calling HistorySelect(0, INT_MAX).

As an experiment, I cut all the calls to history necessary for the trading logic. The load on the CPU has decreased very much.

Generally speaking, if there are 20 robots, calling them all over the history means that you can cause a disaster with only one terminal. About several Terminals we can't even talk about.

And it feels like synchro object is not only history. SymbolInfoTick, CopyTicks and something else, it seems.

Anyway, I can't even get five Terminals running with a dozen robots on each.

Looking at the braking profiler is a bummer.

[Renat Fatkhullin]

fxsaber:

Ordinary

Fully caching the history. But even this requires calling HistorySelect(0, INT_MAX).

As an experiment, I cut off all history calls necessary for the trading logic. The load on the CPU has decreased very much.

Generally speaking, if there are 20 robots, calling them all over the history means that you can cause a disaster with only one terminal. About several Terminals we can't even talk about.

And it feels like synchro object is not only history. SymbolInfoTick, CopyTicks and something else, it seems.

Anyway, I can't even get five Terminals running with a dozen robots on each.

Looking at the braking profiler is a bummer.

No evidence, nor numerical data.

1) How many times per second does each EA make HistorySelect queries?

2) Exactly which functions are slowing down?

3) Logs?

4) What is the principle of the robots?

[Andrey Khatimlianskii]

fxsaber:

All in all, if there are 20 robots, then to address the story in them is to cause a disaster with just one Terminal. Multiple Terminals is out of the question.

Maybe on the contrary - each terminal will support its own synchro object, and there won't be a queue of 20 EAs to it?

Try to run 1 robot on 1 terminal, it will be interesting to see the result.

[fxsaber]

Andrey Khatimlianskii:

Maybe on the contrary - each terminal will support its own sync object and there won't be a queue of 20 EAs to it?

Try to run 1 robot on 1 terminal, it's interesting to see the result.

Unfortunately, the result of this experiment will not answer the question, what to do?

[prostotrader]

fxsaber:

Unfortunately, the result of this experiment will not answer the question, what to do?

Reconsider the concept of the trading robot

Added

I have 3 terminals on real + 1 demo in which I work

Each terminal has 42 robots that use OnBoorEvent from 3 to 4 characters,

Plus every 0.5 sec timer triggers + each robot accesses Global Variables of the terminal,

and it uses 8.34 GB of 32 GB of RAM and 6.7% of CPU

And nothing slows down, except for the TM5 server at the beginning of trading sessions.

[fxsaber]

Renat Fatkhullin:

There is no evidence, nor is there any numerical data.

1) How many times per second does each expert make HistorySelect queries?

2) Which functions exactly are slowing down?

3) Logs?

4) What is the principle behind the robots?

It's very hard for me to answer these questions, because even I myself can't get a handle on what's slowing down. The profiler doesn't even run. Their measurements are cheating, because the lag is already coming from the CPU. Reducing access to environment functions via snapshots and caching, unfortunately, did not give the expected effect. I am waiting for a profiler, which will be able to compile the Expert Advisor.

While I was busy, I found such a mess in the Strategy Tester with the trade history.

void OnTick()
{
  static bool FirstRun = true;
    
  if (FirstRun)
  {
    MqlTick Tick;

    if (SymbolInfoTick(_Symbol, Tick) && Tick.ask)
    {
      MqlTradeRequest Request = {0};
      MqlTradeResult Result;
      
      Request.action = TRADE_ACTION_PENDING;
      Request.type = ORDER_TYPE_BUY_LIMIT;
      Request.symbol = _Symbol;
      Request.volume = 1;
      Request.price = Tick.ask - 10000 * _Point;

      if (OrderSend(Request, Result)) // Выставили отложку.
      {
        Request.action = TRADE_ACTION_DEAL;      
        Request.type = ORDER_TYPE_BUY;
        Request.price = Tick.ask;
        
        FirstRun = !OrderSend(Request, Result); // Открыли позицию.
      }
    }
  }

  HistorySelect(0, INT_MAX); // Результат зависит от этой строки.  
}

// Проверяет наличие ордера в истории торгов.
bool CheckTicket( const long Ticket )
{
  return(HistoryOrderGetInteger(Ticket, ORDER_TICKET) == Ticket);
}

#define  PRINT(A) Print(#A + " = " + (string)(A))

void OnDeinit( const int )
{
  if (HistorySelect(0, INT_MAX))
  {
    PRINT(CheckTicket(4)); // true
    PRINT(CheckTicket(3)); // true
    PRINT(CheckTicket(2)); // false
  }  
}

The result is

        AUDCAD : real ticks begin from 2020.07.15 00:00:00
        2020.07.15 00:01:09   buy limit 1 AUDCAD at 0.84993 (0.94914 / 0.94993)
        2020.07.15 00:01:09   market buy 1 AUDCAD (0.94914 / 0.94993)
        2020.07.15 00:01:09   deal #2  buy 1 AUDCAD at 0.94993 done (based on order #3)
        2020.07.15 00:01:09   deal performed [#2  buy 1 AUDCAD at 0.94993]
        2020.07.15 00:01:09   order performed buy 1 at 0.94993 [#3  buy 1 AUDCAD at 0.94993]
        2020.07.15 23:59:58   position closed due end of test at 0.94646 [#3  buy 1 AUDCAD 0.94993]
        2020.07.15 23:59:58   deal #3  sell 1 AUDCAD at 0.94646 done (based on order #4)
        2020.07.15 23:59:58   deal performed [#3  sell 1 AUDCAD at 0.94646]
        2020.07.15 23:59:58   order performed sell 1 at 0.94646 [#4  sell 1 AUDCAD at 0.94646]
        2020.07.15 23:59:58   order canceled due end of test [#2  buy limit 1 AUDCAD at 0.84993]
        final balance 99999653.00 pips
        2020.07.15 23:59:58   CheckTicket(4) = true
        2020.07.15 23:59:58   CheckTicket(3) = true
        2020.07.15 23:59:58   CheckTicket(2) = false

I have hardly noticed this bug and have been trying to write a replay for over an hour. The code is moronic but it shows the problem. Whether there is something similar not in the Tester but in the Terminal, I don't know.

Search string: Oshibka 013.

ZS b2626 - fixed.

[fxsaber]

prostotrader:

Reconsider the concept of the trading robot

Only one robot that trades all symbols?

[prostotrader]

fxsaber:

Only one robot that trades all symbols?

Different robots, but all built roughly on the same pattern.

There are 42 jobs involved in one terminal at a time, and on three, 126 is about 400 symbols

Added

To repeat (me)

Each robot uses OnBoorEvent from 3 to 4 characters,

plus every 0.5 sec a timer is triggered + each robot accesses theGlobal Variables of the terminal,

8.34 GB of 32 GB of RAM and 6.7% of CPU usage

And nothing slows down, except for the TM5 server (or the Openreach hardware) at the start of trading sessions.

[Dmi3]

prostotrader:

Different robots, but all built roughly according to the same scheme.

There are 42 jobs involved in one terminal at a time, and on three - 126 is about 400 characters

Added

To repeat (me).

Each robot uses OnBoorEvent from 3 to 4 characters,

plus every 0.5 sec a timer is triggered + each robot accesses theGlobal Variables of the terminal,

8.34 GB of 32 GB of RAM and 6.7% of CPU usage

And nothing is slowing down, except for the TM5 server (or Open's iron) at the start of trading sessions.

Here's the weird thing, it's the opposite for me.

My device has 4 terminals, I have reduced the number of Expert Advisors to about 200, I`ve cut all OnBooks, switched back to OnTick and updated my hardware but the problems are the same as with fxsaber.

But there are no lags in the morning in my Opener for a long time already. And what they used to be! Up to 75 seconds sometimes :)