Interaction between MetaTrader 4 and Matlab via CSV Files

Introduction

The Matlab environment's computational power is known to be considerably superior to that of any programming language including MQL4. Wide range of mathematical functions provided by Matlab allows one to perform complex computations fully neglecting theoretical basis of the operations made.

However, the real-time interaction between a trading terminal and Matlab represents a nontrivial task. In this article, I suggest a way to organize data exchange between MetaTrader 4 and Matlab via CSV files.

1. Interworking

Suppose, at incoming of each new bar, MetaTrader 4 must send data about the last 100 bars to Matlab and be responded with their processing results.

To solve this problem, we will need to create an indicator in MetaTrader 4 that would write data into a text file and read the processing results from another text file created by Matlab.

MetaTrader 4 must form its data file at incoming of each new bar. It must try to read the results at each tick, too. In order not to read the result before Matlab updates it, we will have deleted the file containing the result before we form our output file. In this case, the reading attempt will succeed only after Matlab finishes its computation and forms a new file.

Matlab must analyze file attributes created in MetaTrader 4 every second and start processing when its creation time changes. After the processing has finished, the file deleted by MetaTrader 4 before start of data recording is re-created. MetaTrader 4 deletes it successfully, loads new data and awaits respond.

2. Forming an Output Data File

There are many articles devoted to saving data as a files, so I will not labor the point here. I will just put it clear that we write data into 7 columns: “DATE”, “TIME”, “HI”, “LOW”, “CLOSE”, “OPEN”, “VOLUME”. The separating character is “;”. Bar priority is from earlier to later ones, i.e., the line containing the zero bar characteristics must be recorded the last. The file will be provided with a line containing column names. The file name will consist of the symbol name and the timeframe.

#property indicator_chart_window
extern int length = 100;   // The amount of bars sent to be processed
double ExtMap[];           // Chart buffer
string nameData;
int init()
{
   nameData = Symbol()+".txt";         // name of the data file to be sent
   return(0);
}
int start()
{
   static int old_bars = 0;   // remember the amount of bars already known   
   if (old_bars != Bars)      // if a new bar is received 
   {
      write_data();                             // write the data file                              
   }      
   old_bars = Bars;              // remember how many bars are known
   return(0);
}
//+------------------------------------------------------------------+
void write_data()
{
  int handle;
  handle = FileOpen(nameData, FILE_CSV|FILE_WRITE,';');
  if(handle < 1)
  {
    Comment("Creation of "+nameData+" failed. Error #", GetLastError());
    return(0);
  }
  FileWrite(handle, ServerAddress(), Symbol(), Period());                  // heading
  FileWrite(handle, "DATE","TIME","HIGH","LOW","CLOSE","OPEN","VOLUME");   // heading
  int i;
  for (i=length-1; i>=0; i--)
  {
    FileWrite(handle, TimeToStr(Time[i], TIME_DATE), TimeToStr(Time[i], TIME_SECONDS),
                      High[i], Low[i], Close[i], Open[i], Volume[i]);
  }
  FileClose(handle);
  Comment("File "+nameData+" has been created. "+TimeToStr(TimeCurrent(), TIME_SECONDS) );
  return(0);
}

We will not need all these data, of course, but it is always better to have a meaningful file than just a set of columns with unknown figures.

3. Creation of Graphical User Interface (GUI)

So, the file is ready. Let us start Matlab.

We should develop an application that would read text data from file, process and record the results into another file. We will have to create a GUI to specify the file name, view charts and start processing. Let us start now.

To create the GUI, let us start the “GUIDE Quick Start” by typing “guide” in the console or by pressing on the Matlab main panel. In the dialog box appeared, select “Create New GUI” --> “Blank GUI (Default)”. Now we can see the interface for creation a GUI with an empty form. In this form, we will place the following objects: “Edit Text”, “Push Button”, “Static Text”, “Axes”, "Push Button". As a result, it must look somehow like this:

Now we should call the visual property builder for each object and set the properties as follows:

Static Text : HorizontalAlignment – left, Tag – textInfo, String - Info.
Edit Text: HorizontalAlignment – left, Tag – editPath, String – Path select .
Push Button: Tag – pushBrowse, String – Browse.
Axes: Box – on, FontName – MS Sans Serif, FontSize – 8, Tag - axesChart.
Push Button: Tag – pushSrart, String – Start.

By changing the Tag property, we select a unique name for each object. By changing others, we modify the appearance.

when everything is ready, let us launch the interface by pressing “Run”, confirm the interface file saving and saving of the M-file, give a name (for example, “FromTo”), and press “Save”. After that, GUI will be launched and appear as it appears during its work. Matlab generates the M-file to be the basis of our future program and opens it in the embedded editor.

If the appearance does not fit you for some reason, close the working GUI and correct object arrangement using the editor. My distributive, for example, did not show MS Sans Serif correctly. So I had to change it for “Sans Serif”.

4. Building of User Interface

The interface behavior can be programed in M-file Editor using the Matlab language. The skeleton program generated by Matlab represents a list of functions to be called by user when working with the interface objects. The functions are empty, so GUI does not do anything yet. It is our task to fill functions with the necessary contents.

4.1 Programming the Browse Button

First of all, we need access to a file generated by MetaTrader 4, so we will start with the function called by pressing “Browse”.

The name of function called by pressing the button consists of the button name (set by the “Tag” property ) and postfix "_Callback". Let us find function “pushBrowse_Callback” in the file text or just press “Show Functions” on the toolbar and select “pushBrowse_Callback” in the list.

The syntax of Matlab programming language differs from conventional rules of coding in the C and C-like languages. Particularly, there is no need to mark the function body with braces or specify the type of data to be passed into the function, array (vector) indexes start with one, and the comment character is “%”. So, the entire green text above is not a program but a comment made by the Matlab developers for us to be able to grasp the case.

We will need to create a dialog for entering the full name of the file. For this, let us use function “uigetfile”:

% --- Executes on button press in pushBrowse.
function pushBrowse_Callback(hObject, eventdata, handles)
[fileName, filePath] = uigetfile('*.txt'); % receive and the path from the user
if fileName==0          % if it is canceled
    fileName='';        % create an empty name
    filePath='';        % create an empty path
end
fullname = [filePath fileName] % form a new name
set(handles.editPath,'String', fullname); % write in the editPath

“handles” here is a structure that stores descriptors of all objects in our GUI including that of the form, in which we placed them. The structure is passed from a function to another and allows to access to the objects.
“hObject” is a descriptor of the object that has called the function.
“set” helps to set the object value for a certain value and has the following syntax: set(object_descriptor, object_property_name, property_value).

You can find the value of the object properties using the following function: property_value = get(object_descriptor, object_descriptor_name).
But don't forget that the name is a value of the string type, so it must be in single quotes.

The last thing we have to know about objects and their properties. The form, in which we placed the GUI elements, is itself an object placed in the “root” object (its is its descendant). It also has a set of properties that can be modified. Properties can be viewed using the tool named “Object Editor” to be called from the main toolbar of the interface editor. Object “root”, as the term suggests, is the root of graphical objects hierarchy and has no ancestry.

Now let us check what we have as a result. We will now start our GUI by pressing Run on the main toolbar of M-file Editor. Try to click on Browse and select our file. Is it on? Then close the working GUI and go on.

4.2 Programming the Start Button, Chart Drawing

Now we will assign the Start button with calling the function that would read data from the file and show them in a chart.

First, let us create the function itself. We will need the structure of 'handles' objects descriptors as inputs. Having an access to objects, we will be able to read them and to set their properties.

% data reading, chart drawing, processing, storage
function process(handles)
fullname = get(handles.editPath, 'String'); % read the name from editPath
data = dlmread(fullname, ';', 2, 2);    % read the matrix from file
info = ['Last update: ' datestr(now)];  % form an informative message
set(handles.textInfo, 'String',info);   % write info into the status bar
 
high = data(:,1);   % it is now high where the first column of the data matrix is
low = data(:,2);    % d low -- the second
close = data(:,3);  % --/--
open = data(:,4);   %
len = length(open); % the amount of elements in open
 
axes(handles.axesChart); % make the axes active
hold off; % clean axes off before adding a new chart
candle(high, low, close, open); % draw candlesticks (in the current axes)
set(handles.axesChart,'XLim',[1 len]); % set limits for charting

Some explanations:

“dlmread” reads data from the text file with separators and has the following syntax: dlmread(full_file_name, separator, skip_strings, skip_columns);
“length(qqq)” – the larger size of matrix qqq;
”now” – current time and date;
“datestr(now)” – transforms time and date into a text;

You must know, as well, that Matlab provides huge help information with theory and examples.

Let us place our function at the end of the program (it will be easier to find it there) and add its call into “pushStart_Callback”:

% --- Executes on button press in pushStart.
function pushStart_Callback(hObject, eventdata, handles)
process(handles);

Launch it using “Run”, select a file, press "Start", and enjoy the result.

4.3 Saving the Path to a File

Everything is all right now, but it is a bit annoying to permanently click with the mouse selecting a file after having pressed “Browse”. Let us try to save the path once selected.
We will start with reading. The name of a file storing the path will consist of the GUI name and postfix “_saveparam” and have extension of “.mat”.
Function “FromTo_OpeningFcn” is executed directly after the GUI form has been created. We will add there the attempt to read the path from the file. If the attempt fails, the default value will be used.

% --- Executes just before FromTo is made visible.
function FromTo_OpeningFcn(hObject, eventdata, handles, varargin)
guiName = get(handles.figure1, 'Name'); % get the name of our GUI
name = [guiName '_saveparam.mat']       % define the file name
h=fopen(name);                          % try to open the file
if h==-1                                % if the file does not open
    path='D:\';                         % the default value
else
    load(name);                         % read the file    
    fclose(h);                          % close the file
end
set(handles.editPath,'String', path);   % write the name into object "editPath"

Other strings of function “FromTo_OpeningFcn” will be left unaltered.

Let us modify function “pushBrowse_Callback” as follows:

% --- Executes on button press in pushBrowse.
function pushBrowse_Callback(hObject, eventdata, handles)
path = get(handles.editPath,'String'); % read the path from object editPath 
[partPath, partName, partExt] = fileparts(path);    % divide the path into parts
template = [partPath '\*.txt'];                     % create a template of parts
[userName, userPath] = uigetfile(template);         % get the user name and the path from the user
if userName~=0                                      % if "Cancel" is not pressed        
    path = [userPath userName];                     % reassemble the path
end
set(handles.editPath,'String', path);               % write the path into object "editPath"
guiName = get(handles.figure1, 'Name');             % get to know the name of our GUI
save([guiName '_saveparam.mat'], 'path');           % save the path

4.4 Data Processing

As an exemplary process, let us interpolate column “OPEN” by a fourth-order polynomial function.
Let us add the following code at the end of our function, “process”:

fitPoly2 = fit((1:len)',open,'poly4'); % get the polynomial formula
fresult = fitPoly2(1:len); % calculate Y values for X=(from 1 to len)
hold on; % a new chart has been added to the old one
stairs(fresult,'r'); % plot stepwise by color - 'r'- red

Let's try to launch and press “Start”.

If you have approximately the same result as shown above, it is high time start saving data as a file.

4.5 Saving Data as a File

Saving of data isn't more complicated than reading them. The only "nicety" is that vector “fresult” must be counted down, i.e., from the latest to the first one. This is done in order to simplify reading of the file in MetaTrader 4, starting from the zero bar and until the file ends.

Let us complement function “process” by the following code:

[pathstr,name,ext,versn] = fileparts(fullname); % divide the full name
                                                % of the file into parts
newName = [pathstr '\' name '_result' ext];     % re-compose the new file name
fresult = flipud(fresult);  % turn vector fresult
dlmwrite(newName, fresult);    % write in the file

Now, please make sure that the file containing the result has been created, located in the same place where there is the initial file, and has the same name complemented by postfix “_result”.

4.6 Timer Control

This is the most difficult part of the work. We will have to create a timer that would check the MetaTrader 4-formed file creation time every second. If the time changes, function “process” must be launched. The timer stop-start will be performed using “Start”. When GUI opens, we will delete all timers created before.

Let us create a timer by placing the following code within function “FromTo_OpeningFcn”:

timers = timerfind; % find timers
if ~isempty(timers) % if timers are available
    delete(timers); % delete all timers
end
handles.t = timer('TimerFcn',{@checktime, handles},'ExecutionMode','fixedRate','Period',1.0,'UserData', 'NONE');

The above code must be inserted immediately after our previous insertion in this function, i.e., before strings “handles.output = hObject;” and “guidata(hObject, handles);”.

By executing this code, Matlab, immediately after creation of GUI, will check for availability of timers, delete the existing ones and create a new timer. The timer will call function “checktime” every second and pass the list of descriptors “handles” into it. Apart from “handles”, the timer will pass its own descriptor to the function, as well as the structure that contains call time and reason. We cannot influence this, but we must consider this when coding the function to be called by the timer.

You may locate the function itself where you wish. Let it itself write in the Matlab status bar the time when it was called:

% function called by the timer
function checktime(obj, event, handles)
set(handles.textInfo,'String',datestr(now));

At its creation, the timer is stopped, now we should launch it. Let us find function “pushStart_Callback”. Let us comment calling 'process(handles)' placed in it and write timer management into it:

% --- Executes on button press in pushStart.
function pushStart_Callback(hObject, eventdata, handles)
% process(handles);
statusT = get(handles.t, 'Running'); % Get to know the timer status
if strcmp(statusT,'on')     % If it is enabled - 
    stop(handles.t);        % disable it
    set(hObject,'ForegroundColor','b'); % change color of pushStart marking
    set(hObject,'String',['Start' datestr(now)]); % change the button top marking
end     
if strcmp(statusT,'off')    % If it is disabled - 
    start(handles.t);       % enable it
    set(hObject,'ForegroundColor','r');% change color of pushStart marking
    set(hObject,'String',['Stop' datestr(now)]); % change the button top marking
end 

Now let us check how everything works. Let us try to enable and disable the timer using “Start”. If the timer is enabled, the clock above the path input field must function.

It would be more correct to delete the timer using the "X" button at closing of GUI. If you want to do so, add

stop(handles.t) ; % stop the timer
delete(handles.t); % delete the timer

at the beginning of function “figure1_CloseRequestFcn”. This function will be called at closing of GUI. You can access to it from GUI editor:

But, please take into consideration that now, if you press “Run” of the editor without having closed the operating GUI, the old timer will not be deleted while the new one will be created. And next time there will be one more created, etc. You can deal with "unsettled" timers using command “delete(timerfind)” from the Matlab console.

Now, if everything is working fine, we will create a function to check the time of the latest file modification from MetaTrader 4:

% function to be called by the timer
function checktime(obj, event, handles)
filename = get(handles.editPath, 'String'); % get to know the file name 
fileInfo = dir(filename);        % get info about the file
oldTime = get(obj, 'UserData');  % recall the time
if ~strcmp(fileInfo.date, oldTime) % if the time has changed
    process(handles);
end
set(obj,'UserData',fileInfo.date); % save the time
set(handles.pushStart,'String',['Stop ' datestr(now)]); % override the time

Function "dir(full_file_name)" returns a structure that contains the file information (name, date, bytes, isdir). The information about the previous file creation time will be stored in the "Userdata" property of the timer object. Its descriptor is passed to function "checktime" named as obj.

Now, when changing a file created by MetaTrader 4, our program will overwrite the result. You can check this by modifying the file manually (for example, deleting the latest strings) and tracking the changes in the resulting chart or file. Of course, the "Start" button must be pressed at that.

If an extra window containing the chart copy is created during the program's operation, add the following string at the beginning of function "process":

set(handles.figure1,'HandleVisibility','on');

5. Drawing the Results in MetaTrader 4

Now let us return to MetaTrader 4. We have to complement our indicator with a function that would read the result from file and draw it in a chart. The behavior of the program will be described as follows:

1. If a new bar is received: Delete the old result file, Erase the chart, Save the data file.
2. If the result file is readable: Read the file, Draw a chart, Delete the result file.

I will not describe here how the code below works since reading data from file and drawing indicators can be found in other articles. I only notice that the result file here is deleted immediately after it has been put into the chart. So don't worry if you see multiple read error messages.

Read errors occur in two cases:
1. Immediately after a new bar has income, since the result file has not been created yet.
2. Immediately after the result has been read and the chart has been drawn, since the file was deleted in order not to re-read the same data.

Thus, the program keeps its "read error" status for practically all the time. :)

#property indicator_chart_window
#property indicator_buffers 1
#property indicator_width1 2
#property indicator_color1 Tomato
extern int length = 100;   // The amount of bars to be sent for processing
double ExtMap[];           // Chart buffer
string nameData;
string nameResult;

int init()
{
   nameData = Symbol()+".txt";         // the name of the data file to be sent
   nameResult = Symbol()+"_result.txt";// the name of the received file containing results
   SetIndexStyle(0, DRAW_LINE);
   SetIndexBuffer(0, ExtMap);
   return(0);
}
int deinit()
  {
   Comment("");
   return(0);
  }
int start()
{
   static int attempt = 0;    // the amount of attempts to read the result
   static int old_bars = 0;   // remember the amount of the known bars
   
   if (old_bars != Bars)      // if a new bar has income 
   {
      FileDelete(nameResult);                   // delete the result file
      ArrayInitialize( ExtMap, EMPTY_VALUE);    // empty the chart
      write_data();                             // save the data file
      
      old_bars = Bars; return(0);               // nothing should be done this time                           
   }
   //
   int handle_read = FileOpen(nameResult,
                              FILE_CSV|FILE_READ,
                              ';'); // try to open the result file
   attempt++;                       // count the attempt to open
   
   if(handle_read >= 0)             // if the file has opened for reading
   { 
      Comment(nameResult+". Opened with attempt #"+ attempt); // opening report
      read_n_draw(handle_read);  // read the result and draw a chart
      FileClose(handle_read);    // close the file
      FileDelete(nameResult);    // delete the result file
      attempt=0;                 // zeroize the amount of attempts to read
   }
   else                          // if we cannot open the result file
   {
      Comment( "Failed reading "+nameResult+
               ". Amount of attempts: "+attempt+
               ". Error #"+GetLastError()); //Report about failed reading
   }
   old_bars = Bars;              // remember how many bars are known
   return(0);
}
//+------------------------------------------------------------------+
void read_n_draw(int handle_read)
{
   int i=0;
   while ( !FileIsEnding(handle_read)) 
   {
      ExtMap[i] = FileReadNumber(handle_read);
      i++;     
   }
   ExtMap[i-1] = EMPTY_VALUE;
}

 
void write_data()
{
  int handle;
  handle = FileOpen(nameData, FILE_CSV|FILE_WRITE,';');
  if(handle < 1)
  {
    Comment("Failed creating "+nameData+". Error #", GetLastError());
    return(0);
  }
  FileWrite(handle, ServerAddress(), Symbol(), Period());                  // header
  FileWrite(handle, "DATE","TIME","HIGH","LOW","CLOSE","OPEN","VOLUME");   // header
  int i;
  for (i=length-1; i>=0; i--)
  {
    FileWrite(handle, TimeToStr(Time[i], TIME_DATE), TimeToStr(Time[i], TIME_SECONDS),
                      High[i], Low[i], Close[i], Open[i], Volume[i]);
  }
  FileClose(handle);
  Comment("File "+nameData+" has been created. "+TimeToStr(TimeCurrent(), TIME_SECONDS) );
  return(0);
}

Below is my final result. I hope I did not make any mistakes and you will be able to reproduce it.

Conclusion

In this article, we described the way of organizing an interaction between MetaTrader 4 and Matlab via CSV files. This method is neither unique nor optimal. the value of this approach is that it it helps to exchange data arrays without special skills of operating with any programming tools other than MetaTrader 4 and Matlab.