Script Low on Memory, and giving Error loading more than N Indicators

 

Hello fellow developers,

I'm writing a script, which in its functions, needs to read data from several indicators, nearly at the same time.
During this development, I've encountered 2 problems:

The first being a Windows' window popping up telling me MT5 is using up too much memory (even though there are 12GB of RAM available and MT5 is only using ~750MB total), and that it will start closing other applications soon enough, if the memory consumption does not stop increasing.
There are a few memory optimizations that i have done to make the code more memory efficient (such as freeing unsused buffers, and copying small amounts of data from the indicators' buffers at time, etc...). But, this is still not enough avoid the Windows warning.
Is there a way to avoid this Windows "running out of memory" warning?

The second is that, when loading a high number of indicators, MT5 apparently is giving me and error and stopping the script. The error appears with different indicators, so it's not related to a single indicator.
Also, with memory optimization, this error appears far before reaching the ~750MB "memory consumption" Windows warning, which makes me believe it's not related to memory consumption..
Is there a limitation on the number of indicators which can be loaded onto a Symbol or accessed within a Script? 

 

Thank you in advance,
Diogo Seca 

 

Diogo Seca:

...

Is there a way to avoid this Windows "running out of memory" warning?


Is there a limitation on the number of indicators which can be loaded onto a Symbol or accessed within a Script? 


Your code probably need to be improved.
 
Alain Verleyen:

Your code probably need to be improved.

I understand. But bear with me and suppose that I'm doing everything correctly, as far as efficiency goes, to achieve my goal.

I still need to know MQL5 and MT5 limitations, so I can at least redesign my solution, or abandon it altogether.
For example, if you tell me that a symbol or a script can only run up to 64 indicators, then I'll know that for me to access 128 indicators is a task almost impossible to do, or at least, a very inefficient one.

Unfortunately, I cannot find any article, forum post or documentation on these 2 questions.

 
Diogo Seca:

I understand. But bear with me and suppose that I'm doing everything correctly, as far as efficiency goes, to achieve my goal.

I still need to know MQL5 and MT5 limitations, so I can at least redesign my solution, or abandon it altogether.
For example, if you tell me that a symbol or a script can only run up to 64 indicators, then I'll know that for me to access 128 symbols is almost a task impossible to do, or at least, a very inefficient one.

Unfortunately, I cannot find any article, forum post or documentation on these 2 questions.

There is maybe some limitations but I am not aware of it.

If you provide some sample code I could check if you want.

 
Alain Verleyen:

There is maybe some limitations but I am not aware of it.

If you provide some sample code I could check if you want.

Thank you Alain,

My current script source code is quite long and is comprised of several include files.

Therefor, I will create 2 friendly demonstrations for both problems, separately.

 
Alain Verleyen:

There is maybe some limitations but I am not aware of it.

If you provide some sample code I could check if you want.

Alain,

This script will demonstrate problem #2: Limitation on the amount of Indicators / Buffers that a script or terminal can hold, at a given time.

It will attempt to initialize a really big amount of indicators, in several symbols (you can select whether you want MarketWatch symbols or not)

At the moment, the limit number I'm getting seems to variate between 570 and 600 custom indicators.

 

Note: this script requires Examples\Custom Moving Average.ex5, which comes with the standard MT5 instalation.
I've provided you with a copy of mine, in case you don't have it. 

 
I forgot to add this screenshot. These are the errors I'm getting:

example of the output errors
 
Diogo Seca:
I forgot to add this screenshot. These are the errors I'm getting:

Thanks taking time to post this code. So I got the same "problem", but I don't think there is an MT5 limitation, but a resource limitation (memory).
Also, with memory optimization, this error appears far before reaching the ~750MB "memory consumption" Windows warning, which makes me believe it's not related to memory consumption..

I don't know where this value comes from, but from my checking each indicator (handle) is using up to 10 Mb of memory, which seems to me huge. With 7322 Mb of available memory before starting the script I succeed to load 713 handles.
2017.02.21 17:45:08.986    170706 (AUDUSD,H1)    Failed to initialize indicator. Error=4802 Sym=GBPUSD SymIndex=1 IndIndex=213 TotalIndicators=713
 
Alain Verleyen:
I don't know where this value comes from, but from my checking each indicator (handle) is using up to 10 Mb of memory, which seems to me huge. With 7322 Mb of available memory before starting the script I succeed to load 713 handles.

Interesting....

I couldn't reproduce that 713 : 7322 ratio or anything that resembles it: 


Also, it seems that the error invalid EX5 file(12) only appears occasionally. Very strange..

 
Diogo Seca:

Interesting....

I couldn't reproduce that 713 : 7322 ratio or anything that resembles it: 

I am using following code inside "i" loop :

         string msg=StringFormat("I: %i Available = %i Physical = %i Total = %i Used = %i",s*IND_PER_SYMB+i
                                 ,TerminalInfoInteger(TERMINAL_MEMORY_AVAILABLE)
                                 ,TerminalInfoInteger(TERMINAL_MEMORY_PHYSICAL)
                                 ,TerminalInfoInteger(TERMINAL_MEMORY_TOTAL)
                                 ,TerminalInfoInteger(TERMINAL_MEMORY_USED));
         Comment(msg);
Available memory start at 7322 and fall to 0 (before error messages start).

Also, it seems that the error invalid EX5 file(12) only appears occasionally. Very strange..

When you got out of memory, strange things are happening ;-)

Edit: I forgot to say I removed the CheckLoadHistory(), if that makes a difference.
 
Alain Verleyen:

When you got out of memory, strange things are happening ;-)

Indeed! Thank you Alain!

Using your code piece, the increase in indicator count was much slower, but the overall memory consumption was much higher (from under 800MB, now the memory consumption went up to 2.5GB!).
This time, I did get a Windows warning of memory over-consumption. 

So..  

Is there a limitation on the number of indicators which can be loaded onto a Symbol or accessed within a Script? 
Short answer: No.
Long answer: Depends on how much available memory you have.

I'm now wondering whether my PC is making full use of my 16GB RAM, since it starts panicking at close to 8GB of mem.consumption. Perhaps the other 8GB are not being used..
I'll update this post with more info if I find anything MT5\MQL5 related.

Reason: