MT5 RAM memory voraciousness, problems with reading/writing large files - page 4

New comment
 
Maxim Dmitrievsky:

try reading your file like this

https://www.mql5.com/ru/docs/files/filereadarray

I don't have time to open the terminal right now, maybe tomorrow.)

So it reads"from a binary file", where would I get one?

I'll wait for "tomorrow" :)
 
Aleksey Vyazmikin:

So it reads"from a binary file", where do I get one?

I'll wait for "tomorrow" :)

where do you get it from?

 
Maxim Dmitrievsky:

where do you get it from?

In this particular case, the file is glued together from other CSV files in Excel. The files were originally generated by MT5, in this particular case.

 
Aleksey Vyazmikin:

In this particular case, the file is glued together from other CSV files in Excel. It was originally generated by MT5, in this particular case.

So write in the binary from mt5. Is it an array? Just write the array in the file at once and load it later

 

Forum on trading, automated trading systems and trading strategies testing

Features of mql5 language, subtleties and tricks

fxsaber, 2018.04.06 17:08

Probably not many people do that, so here
// Заполнение массива строками из файла - классика
int FileToStrings( const string FileName, string &Str[] )
{
  ArrayResize(Str, 0);

  const int handle = FileOpen(FileName, FILE_READ | FILE_ANSI );
  
  if (handle != INVALID_HANDLE)
  {
    while (!FileIsEnding(handle))
      Str[ArrayResize(Str, ArraySize(Str) + 1) - 1] = FileReadString(handle);
    
    FileClose(handle);
  }
  
  return(ArraySize(Str));
}

// Заполнение массива строками из файла - альтернатива
int FileToStrings2( const string FileName, string &Str[] )
{
  uchar Bytes[];
  
  return(FileLoad(FileName, Bytes) ? StringSplit(CharArrayToString(Bytes), '\n', Str) : 0);
}

void OnStart()
{
  const string FileName = "Test.txt";
  
  string Str[];  
  FileToStrings(FileName, Str);

  string Str2[];  
  FileToStrings2(FileName, Str2);
  
  ArrayPrint(Str);
  ArrayPrint(Str2);
}
I myself use the second option when I need to rasp something. Probably works faster too, haven't tested it.

As a benchmark, possible indicators are

Forum on trading, automated trading systems and trading strategy testing

Scripts: ThirdPartyTicks

fxsaber, 2018.04.18 23:10

Parses (ZIP+CSV) at three million ticks per second. That must be fast.

More than 3GB ZIP (>> 10GB CSV) I process with no problem, so it could definitely work well.

 

The CSV parsing itself should consist of about 10 lines. A whole class is not needed at all in this case.

The task is very simple even for a beginner in programming. What happens to the data after reading from the file is the next task, which only the programmer understands.

Alexey, hire a programmer or start a proper study of programming. Otherwise you will be wasting money on classes you cannot use. It is senseless to try to earn without programming experience.

P.S.

Do not touch Renat. He has nothing to do on this subject. He has to fix the optimization.

 
Maxim Dmitrievsky:

So write in the binary from mt5. Is it an array? Just write the array to the file at once and load it later.

There are many columns, each column is a separate array.

 
fxsaber:

As a benchmark, the following figures are possible

More than 3GB ZIP (>> 10GB CSV) I process without any problems, so it can definitely work quite well.

Thanks. It's clear how to write rows to an array, but I want to work with columns as well. It's supposed that after calling the row number from the array, you need to do some additional parsing to find the columns, but how to do it?

 
Aleksey Vyazmikin:

There are many columns, each column is a separate array.

So it's a 2-dimensional array, a matrix. Write it

 
Roffild:

The CSV parsing itself should consist of about 10 lines. A whole class is not needed at all in this case.

The task is very simple even for a beginner in programming. What happens to the data after reading from the file is the next task, which only the programmer understands.

Alexey, hire a programmer or start a proper study of programming. Otherwise you will be wasting money on classes you cannot use. It is senseless to try to earn without programming experience.

P.S.

Do not touch Renat. He has nothing to do in this subject. He needs to fix the Optimization.

I've been using it for about 3 years already, since MT4. It's convenient, but really slow.

Of course it's easy to read a string, but turning it all into a working cell structure is much harder.

What do you mean I can't use classes? I do, and besides, the metaquote employee wrote the class. And then, I actually hired a person, paid and expected to get a finished product, but I was told that it's OK to be slow... I believed them, but now I've gone overboard with memory.

12345678
New comment