MT5 and speed in action

[fxsaber]

The MT5 is a nimble platform. But there are bottlenecks that negate all efforts at fast trading.

I would like to collect the problems here, discuss and solve them somewhere with my own efforts, somewhere with the help of the Developers.

[fxsaber]

HistorySelect.

This is an insanely expensive feature. And, unfortunately, no amount of caching can make its speed acceptable now.

For example, in battlefield EAs it is important to work with actual data. In particular, that the ticks from Market Watch and CopyTicks are not outdated if possible.

To do this, there are not very good, but forced mechanisms to check the relevance of current price data. One part of this mechanism is to detect situations where a trade on a symbol has passed later than the last tick. This does not happen rarely, so it is important to know if the current tick is still up to date or not.

For this detection, you need to be able to access the history of recent deals. It is done using the HistorySelect in the following schematic way.

void OnTick()
{
  MqlTick Tick;

  if (SymbolInfo(_Symbol, Tick) && HistorySelect(Tick.time, INT_MAX)) // Взяли история торгов со времени текущего тика.
    // Проверяем актуальность тика через сравнение Tick.time_msc и DEAL_TIME_MSC.
}

Unfortunately, such a call of HistorySelect takes 5-30 milliseconds (I measured it myself in Release-EX5). When the Expert Advisor makes several such updates in OnTick (in a good way, it should be done after every pause. For example, after each OrderSend.), then everything becomes insanely expensive/long. HistorySelect can collectively eat up several seconds in a single OnTick.

Dear developers, why is it so expensive? This function can execute instantly with proper implementation.

Please think about introducing such functions to work with history.

HistoryDealsSelect( const int Index, const int Count = WHOLE_ARRAY );  // Из внутренней таблицы сделок взять сделки, начиная с заданного индекса в таблице.
HistoryOrdersSelect( const int Index, const int Count = WHOLE_ARRAY ); // Из внутренней таблицы ордеров взять ордера, начиная с заданного индекса в таблице.

They would completely close the HistorySelect brakes. Because it would solve the problem of getting the latest deals very cheaply. Right now, it's one torment in combat execution.

It is not always possible to control the last deals via OnTradeTransaction. Therefore, quick work with history is urgent.

[fxsaber]

Access to ChartEvent and TradeTransaction events.

An idea and one of the possible implementations has been suggested. So I am just copying it here.

Forum on trading, automated trading systems and trading strategies testing

Errors, bugs, questions

Sergey Dzyublik, 2020.05.20 00:47

Suggestion to the developers.
Please consider adding in MQL a function that allows users to independently call processing of accumulated "messages" in OnChartEvent from custom code.
1) This would allow calling OnChartEvent handling between iterations of a time-consuming calculation, making the user GUI at least somewhat responsive without constructing a vegetable garden of: task pool, data transfer, state synchronization, context saving and restoring...
2) This would allow using OnChartEvent in scripts.

Thanks.

Forum on trading, automated trading systems and trading strategies testing

Bugs, bugs, questions

Ilyas, 2020.05.20 09:09

Are you suggesting to add the function GetNextEvent ?

Forum on trading, automated trading systems and trading strategies testing

Bugs, bugs, questions

Sergey Dzyublik, 2020.05.20 13:39

Not quite, I would rather call this function as HandleNextEvent, a possible signature:

bool HandleNextEvent (ENUM_EVENT_TYPE);

When called, similar to GetNextEvent, it checks if the specified ENUM_EVENT_TYPE is present in the queue,
and if this event is present, it automatically passes control to the user code of the corresponding handler (OnChartEvent, OnTrade, OnTradeTransaction, ... (thanksto fxsaber for the addition)).
Returns true if there was an event in the queue, otherwise returns false.

Possible use case:

//....
for(int i = 0; i < 10^6; ++i){
   // .... Data Calculations

   if((i % 10^3) == 0){
       while(HandleNextEvent(EVENT_TYPE_ALL));
   }
}
//....

Forum on trading, automated trading systems and trading strategy testing

Errors, Bugs, Questions

Sergey Dzyublik, 2020.05.20 14:38

2) That's right. If I am interested in processing of a particular event, not all of the events in the system, it would be nice to be able to process only this type of events, leaving the processing of other events in the usual mode.
3) If HandleNextEvent is called again during processing - call and process. The only thing that can happen is stack overflow, but this is user's and code's problem, not developer's.
4) Events that don't fall under the filter remain in the same sequence and will be called when the user returns control to the system, as usual.

Now we have to make crutches to get to the same TradeTransaction-events inside the running code.

Expert Advisors really need this feature for the combat application.

[prostotrader]

fxsaber:

It is not always possible to monitor recent transactions via OnTradeTransaction. So a quick work with history is relevant.

Please give an example when it is not possible to control?

[fxsaber]

prostotrader:

Please give me an example when there is no control?

Forum on trading, automated trading systems and trading strategies testing

Peculiarities of mql5, tips and tricks

fxsaber, 2020.04.23 19:51

The open/close arrows that MT5 automatically places in real time are based on TradeTransaction events.

I just saw that these events (opening and closing about a dozen positions) did not reach the terminal because of a momentary connection interruption - it was so coincidental, I was sitting at my PC and looked at them firsthand. As a result there are no corresponding arrows.

And, as has sometimes been said here, you can't rely on OnTradeTransaction in combat EAs. Too bad there's no reliable public mechanism for dealing with OrderSendAsync.

And also when several OrderSend are placed inside the code. When one OrderSend depends on the previous one.

[prostotrader]

fxsaber:

Also when placing multiple OrderSend within code. When one OrderSend depends on the previous OrderSend.

If there is a break in communication, nothing will be added to the history either!

I've been usingOrderSendAsync for over 5 years(there were problems with OrderSend back then) + OnTradeTransaction() just in combat mode - no problems!

If there is a break in connection, then it is found out by the wizard which is assigned to each order (in FOREX it is difficult to do because the symbol names are not standardized).

[fxsaber]

prostotrader:

If there is a break in communication, nothing will be added to the history either!

I've been usingOrderSendAsync for over 5 years(there were problems with OrderSend back then) + OnTradeTransaction() just in combat mode - no problems!

You can't help but notice how you've been sinking your teeth into Async+OnTrade for years. You do data actualization through this mechanism and everything else, there's no problem.

Please don't say "you just can't cook" here, at least.

[prostotrader]

fxsaber:

You can't help but notice how you've been pushing for Async+OnTrade for years. You also do data updates through this mechanism and everything else, because there is no problem.

Please don't claim "you just can't cook" here at least.

You are "fighting" for microseconds and at the same time you are writing tons of code, which also eats up time,

and 99% of the time you don't need it.

[fxsaber]

I encourage you to speak out only in a constructive manner.

[A100]

fxsaber:

Access to ChartEvent and TradeTransaction events.

An idea and one of the possible implementations has been suggested. So I'm just copying it here.

Now, in order to get access to those TradeTransaction-events inside the running code, we have to crutches.

You are making crutches in a wrong direction:

Leave in OnXXX functions only copying of events (parameters) in queue and calling the main OnMain function. Move all their code to the duplicate On2XX functions. And call these duplicating On2XX functions from the OnMain in the sequence in which you need, passing them in turn the data from the queues as parameters.

Then run measurements and show the speed gain, and then you can suggest to complement MQL with appropriate functionality. But why add, if you have already done everything yourself here and now?

[fxsaber]

A100:

Leave the OnXXX functions only copying events (parameters) into the queue and calling the main OnMain function. Move all code to duplicate On2XX functions. And call these duplicating On2XX functions from the OnMain in the sequence in which you need, passing them in turn the data from the queue as parameters.

Please give me a rudimentary scheme-code of this idea. At first glance it sounds like a complete misunderstanding.

While executing the OnMain function there is no way of knowing the state of the current real queue. The only workaround to do this is a spyware program as given in the link.