File Operations via WinAPI

Introduction

MQL4 is designed in such a way that even incorrectly written programs are unable to mistakenly delete data from the hard disk. The functions used for file reading and writing operations can work in the following directories only (quote):

• /HISTORY/<current broker> - especially for the FileOpenHistory function;
• /EXPERTS/FILES - common case;
• /TESTER/FILES - especially for testing.

Working with files from other directories is prohibited.

If you still need to work outside the directories (defined for safety reasons), you can call the functions of Windows OS. For this purpose, the functions of API represented in kernel32.dll library are widely used.

File Functions of kernel32.dll

It is based on the script found in the CodeBase under File Operations without Limitations. It is a good example of how functions can be imported into an MQL4 program.

// constants for function _lopen
#define OF_READ               0
#define OF_WRITE              1
#define OF_READWRITE          2
#define OF_SHARE_COMPAT       3
#define OF_SHARE_DENY_NONE    4
#define OF_SHARE_DENY_READ    5
#define OF_SHARE_DENY_WRITE   6
#define OF_SHARE_EXCLUSIVE    7
 
 
#import "kernel32.dll"
   int _lopen  (string path, int of);
   int _lcreat (string path, int attrib);
   int _llseek (int handle, int offset, int origin);
   int _lread  (int handle, string buffer, int bytes);
   int _lwrite (int handle, string buffer, int bytes);
   int _lclose (int handle);
#import

These functions are declared in msdn as obsolete, but they can still be used, see Obsolete Windows Programming Elements. I will give here the description of functions and parameters taken directly from that thread, as they are described by the script author, mandor:

// _lopen  : It opens the specified file. It returns: file descriptor.
// _lcreat : It creates the specified file. It returns: file descriptor.
// _llseek : It places the pointer in the open file. It returns: 
// the new shift of the pointer.
// _lread  : It reads the given number of bytes from the open file. 
// It returns: the number of the read bytes; 0 - if it is the end of the file.
// _lwrite : It writes the data from buffer into the specified file. It returns: 
// the number of written bytes.
// _lclose : It closes the specified file. It returns: 0.
// In case of unsuccessfully completion, all functions return the value of 
// HFILE_ERROR=-1.
 
// path   : String that defines the path and the filename.
// of     : The way of opening.
// attrib : 0 - reading or writing; 1 - only reading; 2 - invisible, or 
// 3 - system file.
// handle : File descriptor.
// offset : The number of bytes, by which the pointer shifts.
// origin : It indicates the initial point and the shifting direction: 0 - 
// forward from the beginning; 1 - from the current position; 2 - backward from the end of the file.
// buffer : Receiving/writing buffer.
// bytes  : The number of bytes to read.
 
// Methods of opening (parameter 'of'):
// int OF_READ            =0; // Open file for reading only
// int OF_WRITE           =1; // Open file for writing only
// int OF_READWRITE       =2; // Open file in the read/write mode
// int OF_SHARE_COMPAT    =3; // Open file in the mode of common 
// shared access. In this mode, any process can open this given 
// file any amount of times. At the attempt to open this file in any other
// mode, the function returns HFILE_ERROR.
// int OF_SHARE_DENY_NONE =4; // Open file in the mode of common access 
// without disabling the reading/writing by another process. At the attempt to open 
// this file in the mode of OF_SHARE_COMPAT, the function returns HFILE_ERROR.
// int OF_SHARE_DENY_READ =5; // Open file in the mode of common access with 
// disabling the reading by another process. At the attempt to open this file 
// with the flags of OF_SHARE_COMPAT and/or OF_READ, or OF_READWRITE, the function 
// returns HFILE_ERROR.
// int OF_SHARE_DENY_WRITE=6; // The same, but with disabling the writing.
// int OF_SHARE_EXCLUSIVE =7; // Disable for this current and for all other processes 
// to access to this file in the modes of reading/writing. The file in this mode can be 
// opened only once (with the current process). All other attempts 
// to open the file will fail.

Function "Reading from File"

Let us consider the function intended for reading from file. Its only parameter is a string variable that contains the file name. The imported function of _lopen(path,0) returns the pointer at an open file and, for its tasks, is very similar to function FileOpen() in MQL4.

//+------------------------------------------------------------------+
//|   read the file and return a string with its contents            |
//+------------------------------------------------------------------+
string ReadFile (string path) 
  {
    int handle=_lopen (path,OF_READ);           
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return ("");
      }
    int result=_llseek (handle,0,0);      
    if(result<0) 
      {
        Print("Error placing the pointer" ); 
        return ("");
      }
    string buffer="";
    string char1="x";
    int count=0;
    result=_lread (handle,char1,1);
    while(result>0) 
      {
        buffer=buffer+char1;
        char1="x";
        count++;
        result=_lread (handle,char1,1);
     }
    result=_lclose (handle);              
    if(result<0)  
      Print("Error closing file ",path);
    return (buffer);
  }

Function _lseek() has its analog in MQL4, too. It's FileSeek(). Function _lclose is used for closing files, like function FileClose(). The only new function is _lread(handle, buffer, bytes) that reads from the given file (the pointer at which must be preliminarily received by the function of _lopen()) into the variable 'buffer' the given number of bytes. You should use a string constant of the necessary length as the 'buffer ' variable. In this example, we can see:

    string char1="x";
    result=_lread (handle,char1,1);

- the string constant 'char' is given that has the length of one, i.e., it allows to read only one byte into it. At the same time, the value of this constant doesn't matter: it can be both "x", "Z" and even " " (the space character). You will not be able to read more bytes into it than it was initially defined for this constant. In this case, the attempts to read 2 or more bytes won't be successful. Besides, the result of function _lread() is the number of bytes really read. If the file is 20 bytes large and you try to read in a 30-byte long variable more than 20 bytes, the function will return 20. If we consecutively apply this function, we will move along the file reading one block of files by another. For example, a file is 22 bytes large. We start to read it by blocks of 10 bytes. Then, after two calls to function __lread(handle, buff, 10), to bytes at the end of the file will remain unread.

At the third call, __lread(handle, buff, 10) will return 2, i.e., the last 2 bytes will be read. At the fourth call, the function will return zero value - no byte is read, the pointer is at the end of the file. It is this fact that underlies the procedure of reading characters from a file in the cycle:

    while(result>0) 
      {
        buffer=buffer+char1;
        char1="x";
        count++;
        result=_lread (handle,char1,1);
     }

As long as the result (the number of bytes read) is more than zero, function _lread(handle, char1, 1) is called in cycles. As you can see, there is nothing complicated in these functions. The value of the read character is saved in the variable named char1. This character is written from the string variable 'buffer' at the next iteration. Upon completion of the operations, the user-defined function ReadFile() returns the contents of the file read in this variable. As you can see it doesn't create any difficulties.

Function "Writing to File"

Writing is, in a sense, even easier than reading. You should open a file and write into it a byte array using function _lwrite (int handle, string buffer, int bytes). Here, handle is a file pointer obtained by function _lopen(), parameter 'buffer' is a string variable, parameter 'bytes' shows how many bytes should be written. Upon writing, the file is closed with function _lclose(). Let's consider the author's function WriteFile():

//+------------------------------------------------------------------+
//|  write the buffer contents to the given path                     |
//+------------------------------------------------------------------+
void WriteFile (string path, string buffer) 
  {
    int count=StringLen (buffer); 
    int result;
    int handle=_lopen (path,OF_WRITE);
    if(handle<0) 
      {
        handle=_lcreat (path,0);
        if(handle<0) 
          {
            Print ("Error creating file ",path);
            return;
          }
        result=_lclose (handle);
     }
    handle=_lopen (path,OF_WRITE);               
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return;
      }
    result=_llseek (handle,0,0);          
    if(result<0) 
      {
        Print("Error placing pointer"); 
        return;
      }
    result=_lwrite (handle,buffer,count); 
    if(result<0)  
        Print("Error writing to file ",path," ",count," bytes");
    result=_lclose (handle);              
    if(result<0)  
        Print("Error closing file ",path);
  }

However, we should perform error checking. First of all, we try to open a file for writing:

    int handle=_lopen (path,OF_WRITE);

(function _lopen() is called with parameter OF_WRITE).

If the attempt fails (handle < 0), it tries to create a file with the specified name:

        handle=_lcreat (path,0);

If this function, too, returns a negative pointer, then function WriteFile() will be truncated. The resting code in this function is clear without further explanations. The simplest start() function allows you to check how script File_Read_Write.mq4 works.

//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
    string buffer=ReadFile("C:\\Text.txt");
    int count=StringLen(buffer);
    Print("Bytes counted:",count);
    WriteFile("C:\\Text2.txt",buffer);   
//----
   return(0);
  }
//+------------------------------------------------------------------+

Note that the back slash ("\") is written twice, though it must be once. The matter is that some special characters, like line feed character ("\n") or tabulation character ("\t") must be written using the back slash. If you forget this, then you may have problems specifying the path in the test variable during the program execution.

Please write two consecutive back slashes in string constant, not only one. In this case, the compiler will unambiguously accept it correctly.

Everything works, but there is one spoon of tar in this barrel of honey: script function ReadFile() operates very slowly for large files.

The reason for such slow reading of the file is in that we read the information one by one byte (character). You can see in the figure above that the file sized as 280 324 bytes required 103 seconds. This time is taken by reading 1 character 280 324 times. You can independently check the working time for script File Read Write.mq4 attached to the article. How can we accelerate reading from file? The answer is forced upon us - the function must read, say, 50 characters, not by one character at a time. Then the amount of _lread() function calls will be 50 times reduced. Therefore, the reading time must be reduced 50 times, too. Let's check it.

New Function of Reading a File in Blocks 50 Bytes Each

Change the code naming the new version as xFiles,mq4. Compile it and launch for execution. It will be recalled here that importing functions from DLLs must be enabled in the settings (Ctrl+O).

Thus, the execution time of the modified script xFiles.mq4 made 2047 milliseconds, which is approximately 2 seconds. Divide 103 seconds (the execution time of the initial script) by 2 seconds and obtain 103 / 2 = 51.5 times. Thus, the program execution time has really changed approximately 50 times, as it was expected to be. How was the code modified to achieve this?

The changes are small:

string ReadFile (string path) 
  {
    int handle=_lopen (path,OF_READ);
    int read_size = 50;           
    string char50="x                                                 ";
 
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return ("");
      }
    int result=_llseek (handle,0,0);      
    if(result<0) 
      {
        Print("Error placing the pointer" ); 
        return ("");
      }
    string buffer="";
    int count=0;
    int last;
    
    result=_lread (handle,char50,read_size);
    int readen;
    while(result>0 && result == read_size) 
      {
        buffer=buffer + char50;
        count++;
        result=_lread (handle,char50,read_size);
        last = result;
     }
    Print("The last read block has the size of, in bytes:", last);
    char50 = StringSubstr(char50,0,last);
    buffer = buffer + char50;    
    result=_lclose (handle);              
    if(result<0)  
      Print("Error closing file ",path);
    return (buffer);
  }

Please note that the string variable 'char50' is now initialized by the constant of 50 characters (character "x" and 49 spaces).

Now we can perform reading from file in such a manner that to read 50 bytes (characters) at a time into this variable:

result=_lread (handle,char50,read_size);

Here: read_size = 50. Of course, it is not likely that the size of the file to be read will always be a multiple of 50 bytes, which means that there will be a time when the value of this function execution will be other than 50. This is a signal to stop the cycle. The last block of characters, read to the variable, will be cut to the amount of bytes really read.

You can change the size of the buffer to be read to the size of N using function lread(), but don't forget to make two modifications:

1. set the value of read_size for 'N';
2. initialize string variable 'char50' of the constant length of N (N<256).
   

Well, we have accelerated the reading operation. The last task remained is processing the error of non-existing path when trying to write a file. In function WriteFile(), an attempt is made to create a file, but the situation is not processed, in which there is no folder containing the path to the file name. So we need another function -

Folder Creating Function

The folder creating function is also available in kernel32.dll - CreateDirectory Function. It should only be noted that this function tries to create only the bottommost folders, it does not create all in-between folders on the path, if they aren't present. For example, if we try use this function to create folder "C:\folder_A\folder_B", the attempt will succeed only if the path of "C:/folder_A" has already existed before the function call. Otherwise, folder_B will not be created. Let's add the new function to the import section:

#import "kernel32.dll"
   int _lopen  (string path, int of);
   int _lcreat (string path, int attrib);
   int _llseek (int handle, int offset, int origin);
   int _lread  (int handle, string buffer, int bytes);
   int _lwrite (int handle, string buffer, int bytes);
   int _lclose (int handle);
   int CreateDirectoryA(string path, int atrr[]);
#import

The first parameter contains the path to create a new folder, while the second parameter, atrr[], serves for specifying permissions for the folder to be created and must be of the _SECURITY_ATTRIBUTES type. We won't provide any information for the second parameter, but just pass the empty array named 'int'. In this case, the folder to be created will inherit all permissions from the parent folder. However, before trying to apply this function, we will have to perform such operation as -

Breaking Apart the Path

Indeed, let's have to create a 4th level folder, for example, like this:

"C:\folder_A\folder_B\folder_C\folder_D"

Here we will call the folder_D a 4th level folder, since there are three levels of folders above it. Disk 'C:' contains "folder_A", folder "C:\folder_A\" contains "folder_B", folder "C:\folder_A\folder_B\" contains "folder_C"; and so on. It means we have to break apart the entire path to the file into array of subfolders. Let's name the necessary function as ParsePath():

//+------------------------------------------------------------------+
//| break apart the path  into an array of subdfolders               |
//+------------------------------------------------------------------+
bool ParsePath(string & folder[], string path)
   {
   bool res = false;
   int k = StringLen(path);
   if (k==0) return(res);
   k--;
 
   Print("Parse path=>", path);
   int folderNumber = 0;
//----
   int i = 0;
   while ( k >= 0 )
      {
      int char = StringGetChar(path, k);
      if ( char == 92) //  back slash "\"
         {
         if (StringGetChar(path, k-1)!= 92)
            {
            folderNumber++;
            ArrayResize(folder,folderNumber);
            folder[folderNumber-1] = StringSubstr(path,0,k);
            Print(folderNumber,":",folder[folderNumber-1]);
            }
         else break;         
         }
      k--;   
      }
   if (folderNumber>0) res = true;   
//----
   return(res);   
   }   
//+------------------------------------------------------------------+

The delimiter between folders is the '\' character that has the value of 92 in the ANSI coding. The function gets 'path' as its argument and fills out the array named folder[], passed to the function, with the names of paths found, starting with the lowest one and closing with the highest one. In our example, the array will contain the following values:

folder[0] = "C:\folder_A\folder_B\folder_C\folder_D";
folder[1] = "C:\folder_A\folder_B\folder_C";
folder[2] = "C:\folder_A\folder_B";
folder[3] = "C:\folder_A";
folder[4] = "C:";

If we want to write a file named as "C:\folder_A\folder_B\folder_C\folder_D\test.txt", we can divide the specified path into file name test.txt and the structure of subfolders "C:\folder_A\folder_B\folder_C\folder_D" that contains this file. If the program fails creating a file on this path, first of all, it should be tried to create a folder of the lowest level, C:\folder_A\folder_B\folder_C\folder_D".

If the attempt to create this folder fails, too, most probably, the parent folder of "C:\folder_A\folder_B\folder_C" is absent. So we will create folders of higher and higher levels until we get a message about the successful completion of function CreateDirectoryA(). This is why we need a function that would fill out string array 'folder[]' with folder names in the ascending order. The very first zero index contains the folder of the lowest level, the root directory is in the latest array index.

Now we can assemble the function itself that creates all necessary in-between folders on the given path:

//+------------------------------------------------------------------+
//|  It creates all necessary folders and subfolders                 |
//+------------------------------------------------------------------+
bool CreateFullPath(string path)
   {
   bool res = false;
   if (StringLen(path)==0) return(false);
   Print("Create path=>",path);
//----
   string folders[];
   if (!ParsePath(folders, path)) return(false);
   Print("Total subfolders:", ArraySize(folders));
   
   int empty[];
   int i = 0;
   while (CreateDirectoryA(folders[i],empty)==0) i++;
   Print("Create folder:",folders[i]);
   i--;
   while (i>=0) 
      {
      CreateDirectoryA(folders[i],empty);
      Print("Created folder:",folders[i]);
      i--;
      }
   if (i<0) res = true;   
//----
   return(res);
   }

Now it remains only to make a small change in function WriteFile() considering the possibility to create a new folder.

//+------------------------------------------------------------------+
//|  write the buffer contents to the given path                     |
//+------------------------------------------------------------------+
void WriteFile (string path, string buffer) 
  {
    int count=StringLen (buffer); 
    int result;
    int handle=_lopen (path,OF_WRITE);
    if(handle<0) 
      {
        handle=_lcreat (path,0);
        if(handle<0) 
          {
            Print ("Error creating file ",path);
            if (!CreateFullPath(path))
               {
               Print("Failed creating folder:",path);
               return;
               }
            else handle=_lcreat (path,0);   
          }
        result=_lclose (handle);
        handle = -1;
     }
    if (handle < 0) handle=_lopen (path,OF_WRITE);               
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return;
      }
    result=_llseek (handle,0,0);          
    if(result<0) 
      {
        Print("Error placing the pointer"); 
        return;
      }
    result=_lwrite (handle,buffer,count); 
    if(result<0)  
        Print("Error writing to file ",path," ",count," bytes");
    result=_lclose (handle);              
    if(result<0)  
        Print("Error closing file ",path);
    return;        
  }

The logic of how the modified function works is given in figure below.

Please note that after the newly created file has been closed, we set the file descriptor variable 'handle' for a negative value.

        result=_lclose (handle);
        handle = -1;

This is done in order to check the value of 'handle' one line below and open the file for reading only if the first opening failed.

    if (handle < 0) handle=_lopen (path,OF_WRITE);

This will allow us to avoid situations where multiple files are opened by mistake and then left without being closed. In such cases, the operating system informs about exceeding the maximal permitted amount of open files and does n't allow us to open new files.

Let's modify function start() to check the new features:

//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
    int start = GetTickCount();
    string buffer=ReadFile("C:\\Text.txt");
 
    int middle = GetTickCount();
    int count=StringLen(buffer);
 
    Print("Bytes read:",count);
 
    WriteFile("C:\\folder_A\\folder_B\\folder_C\\folder_D\\Text2.txt",buffer);   
    int finish = GetTickCount();
    Print("File size is ",count," bytes. Reading:",(middle-start)," ms. Writing:",(finish-middle)," ms.");
//----
   return(0);
  }
//+------------------------------------------------------------------+

and launch script xFiles.mq4 for execution.

Conclusion

It is not very difficult to use the functions of WinAPI, but you should remember the reverse side of leaving the "sandbox":

Before launching an unknown executable application with the extension of ex4 (without MQL4 source code), which requires to import functions from external DLLs, please think of possible consequences.