MT5 RAM memory voraciousness, problems with reading/writing large files - page 2
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
I've run into this problem before. I did a lot of fixes. I don't remember everything - it was a one-time task,
but try settingArrayResize(arrRead_01,arrSize); the 3rd parameter here
intreserve_size=0// reserve size value (excess )
and experiment.
Thank you, but all the lags and memory consumption, judging by my experiments, occur in the class for reading information from the file. Maybe something can be adjusted there?
Judging by the speed of memory reallocation, arrays of strings are stored as arrays of objects, not pointers to them, hence the brakes. We need the developers to clarify this issue, the solution depends on it.
Then, the parser creates a local array of cells from the string and then copies them into allocated memory, which is an unnecessary crutch, and a significant one at that.
If as an array of objects - then, depending on frequency and number of updates of cells in the problem, probably would be cheaper in time not to parse the file at loading, and to store xw-file as an array of file lines and update data on the fly (ie, parse the string at each access to the cell). In any case, the parser needs to be rewritten, it is extremely inefficient and does not support quoted cells and is only suitable for importing numeric tables.
Judging by the speed of memory reallocation, arrays of strings are stored as arrays of objects, not pointers to them, hence the brakes. We need developers to clarify this question, the solution depends on it.
Let's try to call the developer,@Renat Fatkhullin- can you clarify the situation?
Then, the parser creates a local array of cells from a string and then copies them into allocated memory - an unnecessary crutch, and a significant one at that.
How can we get rid of it?
If as arrays of objects, depending on frequency and number of updates of cells in the problem, probably, it would be cheaper in time not to parse file at loading, and to store xw-file as array of file lines and update data on the fly (i.e. parse string on every access to the cell). In any case, the parser needs to be rewritten, it is extremely inefficient and doesn't support quoted cells and is only good for importing numeric tables.
"Array of file lines" - what do you mean by that? If it's just about creating an array to hold all the strings, then as far as I understand, the string length has a character limit, no?
The reading class was written by an MQ employee, I thought it was all intelligently written there.
Parser reads text correctly, I don't agree with you here - the previously attached script confirms it.
Let's try to call the developer,@Renat Fatkhullin- can you clarify the situation?
How can you get rid of it?
"Array of file strings" - what do you mean? If it's just a question of creating an array where all the strings will be written, then as far as I understand, the string length has a character limit, no?
The reading class was written by an MQ employee, I thought it was all intelligently written there.
Parser correctly reads text, I disagree with you - the script previously attached confirms it.
It correctly reads files where there are no quoted strings, inside which there is a delimiter character. Try to read 60;""sample;string"" by it. The output should contain 2 cells: [60] and [sample;string]. You will probably get 3 - [60] ["sample;string"]. (HH besides, qusv allows hyphenated strings :) )
I know how I'd get rid of this in C or pluses - allocate an array of string pointers first and fill it by parsing the string. There are no pointers in mcl, I don't understand how to approach this task. Hopefully Renat can clarify.
"An array of file strings" - what do you mean by that? If it's just about creating an array that will contain all strings, then as far as I understand, the string length has a character limit, no?
I mean, read the file line by line and store each original line of the file in an array without parsing it. When accessing string by format (row, column), take string row, parse it on the fly and give column value, caching parsing result at the same time.
Here is another possible solution. When reading a file, do a parsing - save for each line an array of two integer values: the index of the character that starts the value of the cell in the line, and the length of that substring.
For example:
строка в файле : 54;345;12;12345
индекс символа : 0 3 7 10
длина подстроки: 2 3 2 5
It correctly reads files where there are no quoted strings with a delimiter character inside. Try reading 60;""sample;string"" with it. The output should be 2 cells: [60] and [sample;string]. You will probably get 3 - [60] ["sample;string"]. (HH besides, qusv allows hyphenated strings :) )
Oh, I see, I didn't know about such subtleties of CSV standard. Thanks for enlightening me about the peculiarities!
I know how I'd get rid of it in C or pluses - I'd allocate an array of string pointers first and fill it in, parsing the string. There are no pointers in mcl, I don't understand how to approach this task. Hopefully Renat can clarify.
Hopefully Renat can clarify.
I mean, read the file line by line and store each original line of the file in an array without parsing it. When accessing the string by format (row, column), take the string row, parse it on the fly and return the value of column, caching the result of parsing.
Yes, I think it would be optimal in time, even the cache is not really needed, I guess. Can you help and make appropriate edits to the class to implement this approach?
Here is another possible solution. When reading a file, do a parsing - save for each line an array of two integer values: the index of the character that starts the value of the cell in the line, and the length of that substring.
For example:
строка в файле : 54;345;12;12345
индекс символа : 0 3 7 10
длина подстроки: 2 3 2 5
Well yes, that's about the approach I wrote about above today - calculate what's what first, and then fill in the array. But I have no idea how to implement it :(
WriteArray / Read are fast, max size up to 300 mb, everything is very fast, does not consume RAM
Why is there so much code for reading/writing, it's all done in 4 lines
WriteArray / Read are fast, max size up to 300 mb, everything is very fast, does not consume RAM
Why is there so much code for reading/writing, it's all done in 4 lines