Analyzing trading results using HTML reports

Introduction

If a trader is trying to attract investors, it is very likely that they will want to check his or her trading results. So a trader should be able to present trading history to demonstrate the results. The MetaTrader 5 allows saving the trading history to a file (Toolbox — Trading tab — context menu — Report). A report can be saved as XLSX (to be analyzed in Microsoft Excel) and as HTML files which can be viewed in any browsers. The second option is obviously more popular, since some users may not have Excel, while everyone has a browser. Therefore, an HTML report with trading results would be more suitable for a potential investor.

In addition to standard metrics available in such reports, you may want to calculate your own values using trading data, which can be extracted from a report. In this article, we will consider the methods for extracting data from HTML reports. First, we will analyze the structure of reports and then we will write a function to parse them. The name of the report will be passed to the function. Then the function will return a structure with the relevant data. This structure will enable the direct access to any report value. Using this structure, you will be able to easily and quickly generate your own reports with any desired metrics.

In addition to the trading reports, MetaTrader 5 allows saving Expert Advisor testing and optimization reports. Similarly to trading history, a testing report can be saved in two formats: XLSX and HTML, while the optimization report is saved in XML.

In this article we consider the HTML testing report, the XML optimization report and the HTML trading history report.

HTML and XML files

An HTML file is actually a text file consisting of text (displayed data) and tags, which indicate how the data should be displayed (Fig. 1). Any tag starts with the character "<" and ends with ">". For example, the tag <br> means that the text following it should begin with a new line, while <p> means that the text should start with a new paragraph (not only a new line, but after an empty line). Additional attributes can be located inside tags, for example <p color="red"> means that the text following the tag should start with a new paragraph and be written in red. 

Fig. 1. A fragment of the HTML file opened in Notepad

In order to cancel the tag action, a closing tag is used, which is similar to the opening one and additionally has a slash "/" at the beginning. For example, </p> is the closing tag for a paragraph. Some tags are used without closing counterparts, such as <br>. Some tags can optionally be used with a closing tag. A new paragraph can be started without closing the previous one. However, if a color attribute is used inside the tag, like in the above example with <p>, a closing tag must be used to cancel further text coloring. There are tags for which closing is required, such as <table>. A table should always end with a closing tag </table> . The table consists of rows indicated by the opening <tr> and the closing </tr> tags. Cells inside the rows are defined with <td> and </td>. Sometimes cells are tagged with <th> (header cell). One and the same table may have both cells tagged with <td> and those with <th>. Row and cell closing tags are required.

Currently, most of html attributes are practically not used. One "style" attribute is used instead, in which the element appearance is described. For example: <p style="color: red"> denotes a paragraph in red, but this is also not a very popular method. The most common way is to use the "class" attribute, in which only the style class name is specified, such as <p class="small">, while the class itself (style description) is located at the beginning of the HTML document or in a separate CSS file (Cascading Style Sheet). 

XML files (Fig. 2) are very similar to HTML. The main difference is that HTML supports a strictly limited set of tags, while XML allows the expansion of the tag set and the addition of custom tags.

Fig. 2. A fragment of the XML file opened in Notepad

The main purpose of HTML is the data display, that is why it has a standard set of tags. Due to this HTML documents have almost the same appearance in different browsers. The main purpose of XML is to save and pass data, therefore the possibility of arbitrary data structuring is important, while the user working with this file must understand the purpose of doing so and the data that needs to be extracted.

String functions or regular expressions

Usually, regular expressions are used to extract data from text. In the CodeBase, you can find the RegularExpressions library for working with the expressions. You may also check the article describing how to use them: "Regular expressions for traders". Of course, regular expressions constitute a very powerful and convenient tool for parsing text data. If you often have to deal with data parsing, while performing different tasks, this definitely requires the use of regular expressions.

However, there are a couple of drawbacks of regular expressions: you need to study the expressions properly before using them. The entire idea related to regular expressions is quite extensive. You cannot use them only by "checking the reference when needed". You have to thoroughly study the theory and have practical skills. I think the way of thinking required for the use of regular expressions differs greatly from the one necessary for solving ordinary programming tasks. Even if one has the ability to use regular expressions, difficulties may arise when switching to working with regular expressions and back. 

If data parsing tasks are not difficult and not frequent, you may use standard string functions. Moreover, there is an opinion that string functions operate faster. Most probably, speed depends on the type of the task and conditions for applying regular expressions. However, string functions provide a good solution for the data parsing task.

Of all the string functions available in language reference, we will only need a few: Stringring(), StringSubstr(), StringReplace(). We may additionally need some very simple functions, such as StringLen(), StringTrimLeft(), StringTrimRight().

Tester report

The largest data volume is contained in the tester report, so we will start with it. This will enable us to understand the task and deal with all difficulties which may arise. When creating this article, I used the ExpertMACD EA testing report with default parameters (the report is attached below). Open the report in the browser to see what we are dealing with (Fig. 3).

Fig. 3. Testing report in HTML, opened in browser. Red lines with blue text show the main report sections

The report data are divided into several sections: report name, parameters, results, charts, orders deals, total.  

Each browser has a command for viewing page code: right click on a page to open the context menu and select "View page source" or a similar command. 

From the visual inspection of the source code we can find out that all the report data are arranged in tables (the <table> tag). The tester report has two tables. The first table contains general data: from the report name toll the "Average position holding time" value, i.e. up to the Orders section. The second table contains data concerning orders, deals and the final deposit state. Data of different types are placed within one table using the "colspan" attribute, which unites joins several cells. The names of common parameters and their values are located in different cells of the table, sometimes these cells appear in the same row, sometimes they are located in different rows.

An important moment is the availability of "id" attributes, which are unique identifiers of tags. The id values could be used to find cells containing required data. However, "id" attributes are not used. That is why we will find parameters by counting rows and cells. For example, the "Strategy Tester Report" name is in the first table, first row and first cell. 

The number of orders and deals in the second table will be different in all reports, so we need to find the end of orders and the beginning of deals. A row with one cell follows the deals, and then comes a row with headers — this is an indication for separating data. Now we will not go into detail regarding the number of rows and cells. A convenient way will be presented later.

All data are located in table cells, therefore all we need to do at the initial stage is to extract tables, rows and cells.

Let's analyze trading history and optimization reports. The history report is very similar to the testing report, except for the additional section with the final deposit state (Fig. 4).

Fig. 4. Trading history report in HTML, opened in browser. Red lines with blue text show the main report sections

By examining the trading history HTML code, we can see that cell are defined not only by <td> tags, but also by <th>, which stands for the heading cell. In addition, all data are located in one table.

The optimization report is very simple — it has one data section with one table. From its source code, we see that it is created using the <Table> tag, rows are defined with <Row>, cells are defined with <Cell> (all tags are used alongside appropriate closing tags).

File download

Before we can perform any operations with the report data, we need to read them from the file. Let us use the following function which returns the entire file contents in one string variable:

bool FileGetContent(string aFileName,string & aContent){
   int h=FileOpen(aFileName,FILE_READ|FILE_TXT);
   if(h==-1)return(false);
   aContent="";
   while(!FileIsEnding(h)){
      aContent=aContent+FileReadString(h);
   }
   FileClose(h);
   return(true);
}

The first parameter passed to the function is the name of the report file. The second parameter is the variable which will contain the file contents. The function returns true/false depending on the result.

Extracting the tables

Let us use two structures to locate table data. The structure containing one table row:

struct Str{
   string td[];
};

Each element of the td[] array will contain the contents of one cell.

The structure containing the entire table (all rows):

struct STable{
   Str tr[];
};

The data will be extracted from the report as follows: first we will find the table beginning using the opening tag. Since tags may have attributes, we will only search for the tag beginning: "<table". After finding the beginning of the opening tag, let's find its end, ">". Then search for the table end, i.e. the closing tag "</table>". This is easy, because reports do not have nested tables, i.e. each opening tag is followed by a closing tag.

After finding the table, let's repeat the same for rows using "<tr" for their beginning and "</tr" for their end. Then, in each row we will find the beginning or cells with "<td" and their end with </td>. The task is a bit more complicated with rows, because a cell can have both the tag <td> and <th>. For this reason, we will use the custom TagFind() finction instead of StringFind():

int TagFind(string aContent,string & aTags[],int aStart,string & aTag){
   int rp=-1;
   for(int i=0;i<ArraySize(aTags);i++){
      int p=StringFind(aContent,"<"+aTags[i],aStart);
      if(p!=-1){
         if(rp==-1){
            rp=p;
            aTag=aTags[i];
         }
         else{
            if(p<rp){
               rp=p;
               aTag=aTags[i];
            }
         }      
      }
   }
   return(rp);
}

Function parameters:

• string aContent — the string in which we search;
• string & aTags[] — an array with tags;
• int aStart — the position to start search at;
• string & aTag — the found tag is returned by reference.

Unlike StringFind(), an array is passed to the TagFind() function rather than the searched string. Opening "<" is added to search strings in the function. The function returns the position of the nearest tag.

Using TagFind(), we will consistently search for opening and closing tags. Everything found between them will be collected to an array:

int TagsToArray(string aContent,string & aTags[],string & aArray[]){
   ArrayResize(aArray,0);
   int e,s=0;
   string tag;
   while((s=TagFind(aContent,aTags,s,tag))!=-1 && !IsStopped()){  
      s=StringFind(aContent,">",s);
      if(s==-1)break;
      s++; 
      e=StringFind(aContent,"</"+tag,s);   
      if(e==-1)break;  
      ArrayResize(aArray,ArraySize(aArray)+1);
      aArray[ArraySize(aArray)-1]=StringSubstr(aContent,s,e-s);  
   }
   return(ArraySize(aArray));
}

Function parameters:

• string aContent — the string in which we search;
• string & aTags[] — an array with tags;
• string aArray[] — an array with the contents of all found tags is returned by reference.

An array of searched tags is passed to TagsToArray() only when parsing cells. That is why, for all other cases we will write an analogue of the function with a usual string parameter:

int TagsToArray(string aContent,string aTag,string & aArray[]){
   string Tags[1];
   Tags[0]=aTag;
   return(TagsToArray(aContent,Tags,aArray));
}

Function parameters:

• string aContent — the string in which we search;
• string & aTag — searched tag;
• string aArray[] — an array with the contents of all found tags is returned by reference.

Now let's proceed to the function which can extract the table contents. The string parameter aFileName with the name of the parsed file is passed to the function. A local string variable for the file contents and a local array of structures STable will be used in the function:

STable tables[];
string FileContent;

Obtain the entire report contents using the FileGetContent function:

if(!FileGetContent(aFileName,FileContent)){
   return(true);
}

Here are auxiliary variables:

string tags[]={"td","th"};
string ttmp[],trtmp[],tdtmp[];
int tcnt,trcnt,tdcnt;

Possible options for the cell tags are provided in the 'tags' array. The contents of the tables, rows and cells will be temporarily placed to the string arrays ttmp[], trtmp[] and tdtmp[] respectively. 

Retrieving data from the table:

tcnt=TagsToArray(FileContent,"table",ttmp);
ArrayResize(tables,tcnt);

Loop through all tables and extract rows, then extract cells from rows:

for(int i=0;i<tcnt;i++){
   trcnt=TagsToArray(ttmp[i],"tr",trtmp);
   ArrayResize(tables[i].tr,trcnt);      
   for(int j=0;j<trcnt;j++){         
      tdcnt=TagsToArray(trtmp[j],tags,tdtmp);
      ArrayResize(tables[i].tr[j].td,tdcnt);
      for(int k=0;k<tdcnt;k++){  
         tables[i].tr[j].td[k]=RemoveTags(tdtmp[k]);
      }
   }
} 

Tables, rows and sells are first received to temporary arrays and then the array of cells is copied into a structure. During copying, a cell is filtered by the RemoveTags() function. Table cells can have nested tags. The RemoveTags() deletes them while leaving only required tags.

The RemoveTags() function:

string RemoveTags(string aStr){
   string rstr="";
   int e,s=0;
   while((e=StringFind(aStr,"<",s))!=-1){
      if(e>s){

         rstr=rstr+StringSubstr(aStr,s,e-s);
      }
      s=StringFind(aStr,">",e);
      if(s==-1)break;
      s++;
   }
   if(s!=-1){
      rstr=rstr+StringSubstr(aStr,s,StringLen(aStr)-s);
   }
   StringReplace(rstr,"&nbsp;"," ");
   while(StringReplace(rstr,"  "," ")>0);
   StringTrimLeft(rstr);
   StringTrimRight(rstr);
   return(rstr);
}

Let us consider the RemoveTags() function. The "s" variable is used for the beginning of used data. Its data is first equal to 0, because data can start from the line beginning. The opening angle bracket "<" which means the tag beginning is searched in the "while" loop. When the tag beginning is found, all data from the position specified in the "s" variable up to the found position is copied into the "rstr" variable. After that the end of the tag is searched, that is, the new beginning of useful data. After the loop, if the value of the "s" variable is not equal to -1 (this means that the string ends with useful data, but they have not been copied), the data is copied to the "rstr" variable. At the function end, the space character &nbsp; is replaced with a simple space, while repeated spaces as well as spaces at the beginning and end of the string are deleted.

At this step, we have the "tables" array of structures filled with pure table data. Let us save this array to a text file. While saving, we set numbers for the tables, rows and cells (data is saved to the file 1.txt):

int h=FileOpen("1.txt",FILE_TXT|FILE_WRITE);

for(int i=0;i<ArraySize(tables);i++){
   FileWriteString(h,"table "+(string)i+"\n");
   for(int j=0;j<ArraySize(tables[i].tr);j++){      
      FileWriteString(h,"   tr "+(string)j+"\n");         
      for(int k=0;k<ArraySize(tables[i].tr[j].td);k++){
         FileWriteString(h,"      td "+(string)k+": "+tables[i].tr[j].td[k]+"\n");
      }
   }
}

FileClose(h);

From this file, we can easily understand in what cells the data is located. Below is the file fragment:

table 0
   tr 0
      td 0: Strategy Test report
   tr 1
      td 0: IMPACT-Demo (Build 1940)
   tr 2
      td 0: 
   tr 3
      td 0: Settings
   tr 4
      td 0: Expert Advisor:
      td 1: ExpertMACD
   tr 5
      td 0: Symbol:
      td 1: USDCHF
   tr 6
      td 0: Period:
      td 1: H1 (2018.11.01 - 2018.12.01)
   tr 7
      td 0: Parameters:
      td 1: Inp_Expert_Title=ExpertMACD
   tr 8
      td 0: 
      td 1: Inp_Signal_MACD_PeriodFast=12
   tr 9
      td 0: 
      td 1: Inp_Signal_MACD_PeriodSlow=24
   tr 10
      td 0: 
      td 1: Inp_Signal_MACD_PeriodSignal=9
   tr 11

Structure for the report data

Now we have a bit of boring routine: we need to write a structure corresponding to the report data and fill this structure with data from the tables array. The report is divided into several sections. Therefore, we will use several structures combined into one general structure.

Structure for the Settings section:

struct SSettings{
   string Expert;
   string Symbol;
   string Period;
   string Inputs;
   string Broker;
   string Currency;
   string InitialDeposit;
   string Leverage;
};

The purpose of the structure fields is clear from their names. All the fields of the structure will contain the data exactly as it appears in the report. The list of Expert Advisor parameters will be located in one string variable "Inputs".

Structure for the Results data section:

struct SResults{
   string HistoryQuality;
   string Bars;
   string Ticks;
   string Symbols;
   string TotalNetProfit;
   string BalanceDrawdownAbsolute;
   string EquityDrawdownAbsolute;
   string GrossProfit;
   string BalanceDrawdownMaximal;
   string EquityDrawdownMaximal;
   string GrossLoss;
   string BalanceDrawdownRelative;
   string EquityDrawdownRelative;
   string ProfitFactor;
   string ExpectedPayoff;
   string MarginLevel;
   string RecoveryFactor;
   string SharpeRatio;
   string ZScore;
   string AHPR;
   string LRCorrelation;
   string OnTesterResult;
   string GHPR;
   string LRStandardError;
   string TotalTrades;
   string ShortTrades_won_pers;
   string LongTrades_won_perc;
   string TotalDeals;
   string ProfitTrades_perc_of_total;
   string LossTrades_perc_of_total;
   string LargestProfitTrade;
   string LargestLossTrade;
   string AverageProfitTrade;
   string AverageLossTrade;
   string MaximumConsecutiveWins_cur;
   string MaximumConsecutiveLosses_cur;
   string MaximalConsecutiveProfit_count;
   string MaximalConsecutiveLoss_count;
   string AverageConsecutiveWins;
   string AverageConsecutiveLosses;
   string Correlation_Profits_MFE;
   string Correlation_Profits_MAE;
   string Correlation_MFE_MAE;      
   string MinimalPositionHoldingTime;
   string MaximalPositionHoldingTime;
   string AveragePositionHoldingTime;
};

Structure for data concerning one order:

struct SOrder{
   string OpenTime;
   string Order;
   string Symbol;
   string Type;
   string Volume;
   string Price;
   string SL;
   string TP;
   string Time;
   string State;
   string Comment;
};

Structure for data concerning one deal:

struct SDeal{
   string Time;
   string Deal;
   string Symbol;
   string Type;
   string Direction;
   string Volume;
   string Price;
   string Order;
   string Commission;
   string Swap;
   string Profit;
   string Balance;
   string Comment;
};

Structure for the final deposit state:

struct SSummary{
   string Commission;
   string Swap;
   string Profit;
   string Balance;
};

General structure:

struct STestingReport{
   SSettings Settings;
   SResults Results;
   SOrder Orders[];
   SDeal Deals[];
   SSummary Summary;
};

The SOrder and SDeals structure arrays are used for orders and deals.

Filling the structure with data

This bit of routine work also requires attention. View the previously received text file with the table data and fill in the structure:

aTestingReport.Settings.Expert=tables[0].tr[4].td[1];
aTestingReport.Settings.Symbol=tables[0].tr[5].td[1];
aTestingReport.Settings.Period=tables[0].tr[6].td[1];
aTestingReport.Settings.Inputs=tables[0].tr[7].td[1];

In the last line of the above code example, a value is assigned to the Inputs field, but after that the field will only store data of one parameter. Then other parameters are collected. These parameters are located starting with the row 8, while the first cell in each parameter row is empty. The loop is executed as long as the first cell in the row is empty:

int i=8;
while(i<ArraySize(tables[0].tr) && tables[0].tr[i].td[0]==""){
   aTestingReport.Settings.Inputs=aTestingReport.Settings.Inputs+", "+tables[0].tr[i].td[1];
   i++;
}

Below is the complete testing report parsing function:

bool TestingHTMLReportToStruct(string aFileName,STestingReport & aTestingReport){

   STable tables[];

   string FileContent;
   
   if(!FileGetContent(aFileName,FileContent)){
      return(true);
   }

   string tags[]={"td","th"};
   string ttmp[],trtmp[],tdtmp[];
   int tcnt,trcnt,tdcnt;
   
   tcnt=TagsToArray(FileContent,"table",ttmp);

   ArrayResize(tables,tcnt);
   
   for(int i=0;i<tcnt;i++){
      trcnt=TagsToArray(ttmp[i],"tr",trtmp);
      ArrayResize(tables[i].tr,trcnt);      
      for(int j=0;j<trcnt;j++){         
         tdcnt=TagsToArray(trtmp[j],tags,tdtmp);
         ArrayResize(tables[i].tr[j].td,tdcnt);
         for(int k=0;k<tdcnt;k++){  
            tables[i].tr[j].td[k]=RemoveTags(tdtmp[k]);
         }
      }
   }   
   
   // settings section
   
   aTestingReport.Settings.Expert=tables[0].tr[4].td[1];
   aTestingReport.Settings.Symbol=tables[0].tr[5].td[1];
   aTestingReport.Settings.Period=tables[0].tr[6].td[1];
   aTestingReport.Settings.Inputs=tables[0].tr[7].td[1];
   int i=8;
   while(i<ArraySize(tables[0].tr) && tables[0].tr[i].td[0]==""){
      aTestingReport.Settings.Inputs=aTestingReport.Settings.Inputs+", "+tables[0].tr[i].td[1];
      i++;
   }
   aTestingReport.Settings.Broker=tables[0].tr[i++].td[1];
   aTestingReport.Settings.Currency=tables[0].tr[i++].td[1];  
   aTestingReport.Settings.InitialDeposit=tables[0].tr[i++].td[1];
   aTestingReport.Settings.Leverage=tables[0].tr[i++].td[1];   
   
   // results section
   
   i+=2;
   aTestingReport.Results.HistoryQuality=tables[0].tr[i++].td[1];
   aTestingReport.Results.Bars=tables[0].tr[i].td[1];
   aTestingReport.Results.Ticks=tables[0].tr[i].td[3];
   aTestingReport.Results.Symbols=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.TotalNetProfit=tables[0].tr[i].td[1];
   aTestingReport.Results.BalanceDrawdownAbsolute=tables[0].tr[i].td[3];
   aTestingReport.Results.EquityDrawdownAbsolute=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.GrossProfit=tables[0].tr[i].td[1];
   aTestingReport.Results.BalanceDrawdownMaximal=tables[0].tr[i].td[3];
   aTestingReport.Results.EquityDrawdownMaximal=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.GrossLoss=tables[0].tr[i].td[1];
   aTestingReport.Results.BalanceDrawdownRelative=tables[0].tr[i].td[3];
   aTestingReport.Results.EquityDrawdownRelative=tables[0].tr[i].td[5];
   i+=2;
   aTestingReport.Results.ProfitFactor=tables[0].tr[i].td[1];
   aTestingReport.Results.ExpectedPayoff=tables[0].tr[i].td[3];
   aTestingReport.Results.MarginLevel=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.RecoveryFactor=tables[0].tr[i].td[1];
   aTestingReport.Results.SharpeRatio=tables[0].tr[i].td[3];
   aTestingReport.Results.ZScore=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.AHPR=tables[0].tr[i].td[1];
   aTestingReport.Results.LRCorrelation=tables[0].tr[i].td[3];
   aTestingReport.Results.tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.GHPR=tables[0].tr[i].td[1];
   aTestingReport.Results.LRStandardError=tables[0].tr[i].td[3];
   i+=2;
   aTestingReport.Results.TotalTrades=tables[0].tr[i].td[1];
   aTestingReport.Results.ShortTrades_won_pers=tables[0].tr[i].td[3];
   aTestingReport.Results.LongTrades_won_perc=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.TotalDeals=tables[0].tr[i].td[1];
   aTestingReport.Results.ProfitTrades_perc_of_total=tables[0].tr[i].td[3];
   aTestingReport.Results.LossTrades_perc_of_total=tables[0].tr[i].td[5];
   i++;
   aTestingReport.Results.LargestProfitTrade=tables[0].tr[i].td[2];
   aTestingReport.Results.LargestLossTrade=tables[0].tr[i].td[4];
   i++;
   aTestingReport.Results.AverageProfitTrade=tables[0].tr[i].td[2];
   aTestingReport.Results.AverageLossTrade=tables[0].tr[i].td[4];
   i++;
   aTestingReport.Results.MaximumConsecutiveWins_cur=tables[0].tr[i].td[2];
   aTestingReport.Results.MaximumConsecutiveLosses_cur=tables[0].tr[i].td[4];
   i++;
   aTestingReport.Results.MaximalConsecutiveProfit_count=tables[0].tr[i].td[2];
   aTestingReport.Results.MaximalConsecutiveLoss_count=tables[0].tr[i].td[4];
   i++;
   aTestingReport.Results.AverageConsecutiveWins=tables[0].tr[i].td[2];
   aTestingReport.Results.AverageConsecutiveLosses=tables[0].tr[i].td[4];    
   i+=6;
   aTestingReport.Results.Correlation_Profits_MFE=tables[0].tr[i].td[1];
   aTestingReport.Results.Correlation_Profits_MAE=tables[0].tr[i].td[3];
   aTestingReport.Results.Correlation_MFE_MAE=tables[0].tr[i].td[5];    
   i+=3;
   aTestingReport.Results.MinimalPositionHoldingTime=tables[0].tr[i].td[1];
   aTestingReport.Results.MaximalPositionHoldingTime=tables[0].tr[i].td[3];
   aTestingReport.Results.AveragePositionHoldingTime=tables[0].tr[i].td[5];   
   
   // orders

   ArrayFree(aTestingReport.Orders);
   int ocnt=0;
   for(i=3;i<ArraySize(tables[1].tr);i++){
      if(ArraySize(tables[1].tr[i].td)==1){
         break;
      }   
      ArrayResize(aTestingReport.Orders,ocnt+1);
      aTestingReport.Orders[ocnt].OpenTime=tables[1].tr[i].td[0];
      aTestingReport.Orders[ocnt].Order=tables[1].tr[i].td[1];
      aTestingReport.Orders[ocnt].Symbol=tables[1].tr[i].td[2];
      aTestingReport.Orders[ocnt].Type=tables[1].tr[i].td[3];
      aTestingReport.Orders[ocnt].Volume=tables[1].tr[i].td[4];
      aTestingReport.Orders[ocnt].Price=tables[1].tr[i].td[5];
      aTestingReport.Orders[ocnt].SL=tables[1].tr[i].td[6];
      aTestingReport.Orders[ocnt].TP=tables[1].tr[i].td[7];
      aTestingReport.Orders[ocnt].Time=tables[1].tr[i].td[8];
      aTestingReport.Orders[ocnt].State=tables[1].tr[i].td[9];
      aTestingReport.Orders[ocnt].Comment=tables[1].tr[i].td[10];      
      ocnt++;
   }
   
   // deals
   
   i+=3;
   ArrayFree(aTestingReport.Deals);
   int dcnt=0;
   for(;i<ArraySize(tables[1].tr);i++){
      if(ArraySize(tables[1].tr[i].td)!=13){
         if(ArraySize(tables[1].tr[i].td)==6){
            aTestingReport.Summary.Commission=tables[1].tr[i].td[1];
            aTestingReport.Summary.Swap=tables[1].tr[i].td[2];
            aTestingReport.Summary.Profit=tables[1].tr[i].td[3];
            aTestingReport.Summary.Balance=tables[1].tr[i].td[4];            
         }
         break;
      }   
      ArrayResize(aTestingReport.Deals,dcnt+1);   
      aTestingReport.Deals[dcnt].Time=tables[1].tr[i].td[0];
      aTestingReport.Deals[dcnt].Deal=tables[1].tr[i].td[1];
      aTestingReport.Deals[dcnt].Symbol=tables[1].tr[i].td[2];
      aTestingReport.Deals[dcnt].Type=tables[1].tr[i].td[3];
      aTestingReport.Deals[dcnt].Direction=tables[1].tr[i].td[4];
      aTestingReport.Deals[dcnt].Volume=tables[1].tr[i].td[5];
      aTestingReport.Deals[dcnt].Price=tables[1].tr[i].td[6];
      aTestingReport.Deals[dcnt].Order=tables[1].tr[i].td[7];
      aTestingReport.Deals[dcnt].Commission=tables[1].tr[i].td[8];
      aTestingReport.Deals[dcnt].Swap=tables[1].tr[i].td[9];
      aTestingReport.Deals[dcnt].Profit=tables[1].tr[i].td[10];
      aTestingReport.Deals[dcnt].Balance=tables[1].tr[i].td[11];
      aTestingReport.Deals[dcnt].Comment=tables[1].tr[i].td[12];
      dcnt++;
   }
   return(true);
}

The name of the report file is passed to function. Also the STestingReport structure to be filled in the function is passed by reference.

Note the code parts starting with "Orders" and "Deals" comments. The number of the line with the order list beginning is already defined, while the end of the order list is determined based on a row with a single cell:

if(ArraySize(tables[1].tr[i].td)==1){
   break;
}  

Three rows are skipped after the orders:

// deals
   
i+=3;

After that data on deals is collected. The end of the deal list is determined by the row having 6 cells - this row contains data on the final status of the deposit. Before exiting the loop, the structure of the final deposit state is filled:

if(ArraySize(tables[1].tr[i].td)!=13){
   if(ArraySize(tables[1].tr[i].td)==6){
      aTestingReport.Summary.Commission=tables[1].tr[i].td[1];
      aTestingReport.Summary.Swap=tables[1].tr[i].td[2];
      aTestingReport.Summary.Profit=tables[1].tr[i].td[3];
      aTestingReport.Summary.Balance=tables[1].tr[i].td[4];            
   }
   break;
} 

The structure with the report data is completely ready.

Trading history report

The trading history report can be parsed similarly to the Strategy Tester report, although the final data structure and structures contained therein will be different:

struct SHistory{
   SHistoryInfo Info;
   SOrder Orders[];
   SDeal Deals[];
   SSummary Summary;  
   SDeposit Deposit;
   SHistoryResults Results;
};

The SHistory structure contains the following structures: SHistoryInfo — general account information, arrays with order and deal data structures, SSummary — trading results, SDeposit — final deposit state, SHistoryResults — general values (profit, number of deals, drawdown, etc.).

The full HistoryHTMLReportToStruct() function code for parsing trade reports is attached below. Two parameters are passed to the function:

• string FileName — the name of the report file
• SHistory History — the structure with the trading report data to be filled

Optimization report

The first difference concerning the optimization report is the different file type. The report is saved in ANSI, so we will use another function to read its contents:

bool FileGetContentAnsi(string aFileName,string & aContent){
   int h=FileOpen(aFileName,FILE_READ|FILE_TXT|FILE_ANSI);
   if(h==-1)return(false);
   aContent="";
   while(!FileIsEnding(h)){
      aContent=aContent+FileReadString(h);
   }
   FileClose(h);
   return(true);
}

Another difference is found in tags. Instead of <table>, <tr> and <td>, the following tags are used: <Table>, <Row> and <Cell>. The last main difference is associated with the data structure:

struct SOptimization{
   string ParameterName[];
   SPass Pass[];
};

The structure includes a string array with the names of parameters to optimize (the rightmost report columns) and the array of SPass structures:

struct SPass{
   string Pass;
   string Result;
   string Profit;
   string ExpectedPayoff;
   string ProfitFactor;
   string RecoveryFactor;
   string SharpeRatio;
   string Custom;
   string EquityDD_perc;
   string Trades;
   string Parameters[];
};

The structure includes the fields contained in the report columns. The last field is the string array, which contains the values of parameters under optimization in accordance with their names in the ParameterName() array.

The full code of the OptimizerXMLReportToStruct() function for parsing optimization reports is attached below. Two parameters are passed to the function:

• string FileName — the name of the report file
• SOptimization Optimization — the structure with the optimization report data to be filled

Auxiliary functions

If we arrange all the structures and functions created within this article in one include file, then report parsing will be implemented in three lines of code. 

1. Include the file:

#include <HTMLReport.mqh>

2. Declare the structure:

SHistory History;

3. Call the function:

HistoryHTMLReportToStruct("ReportHistory-555849.html",History);

After that, all report data will be located in the corresponding structure fields, exactly as they are arranged in the report. Although all the fields are of string type, they can be easily used for calculations. For that purpose, we only need to convert the string to a number. However, some of the report fields and appropriate structure fields contain two values. For example, order volume is written by two values "0.1 / 0.1", where the first value is the order volume and the second value represents the filled volume. Some totals also have double values, the main one and an additional value in brackets. For example, the number of trades can be written as "11 (54.55%)" — the actual number and the relative percentage value. Therefore, let us write auxiliary functions to simplify the use of such values.

Functions for extracting individual volume values:

string Volume1(string aStr){
   int p=StringFind(aStr,"/",0);
   if(p!=-1){
      aStr=StringSubstr(aStr,0,p);
   }
   StringTrimLeft(aStr);
   StringTrimRight(aStr);
   return(aStr);
}

string Volume2(string aStr){
   int p=StringFind(aStr,"/",0);
   if(p!=-1){
      aStr=StringSubstr(aStr,p+1);
   }
   StringTrimLeft(aStr);
   StringTrimRight(aStr);
   return(aStr);
}

The Volume1() function retrieves the first volume value from a string like "0.1 / 0.1", Volume2() retrieves the second of the values.

Functions for extracting values double data:

string Value1(string aStr){
   int p=StringFind(aStr,"(",0);
   if(p!=-1){
      aStr=StringSubstr(aStr,0,p);
   }
   StringTrimLeft(aStr);
   StringTrimRight(aStr);
   return(aStr);
}

string Value2(string aStr){
   int p=StringFind(aStr,"(",0);
   if(p!=-1){
      aStr=StringSubstr(aStr,p+1);
   }
   StringReplace(aStr,")","");
   StringReplace(aStr,"%","");
   StringTrimLeft(aStr);
   StringTrimRight(aStr);
   return(aStr);
}

The Value1() function retrieves the first value from a string of type "8.02 (0.08%)", Value2() retrieves the second value, deletes the closing bracket and the percent sign if there is any.

Another useful function which can be needed, is the one for converting data about parameters from the testing report structure. We will use the following structure for converting it into a convenient form:

struct SInput{
   string Name,
   string Value;
}

The Name field is used for the parameter name, Value is used for its value. The value type is known in advance, that is why the Value field has the string type.

The following function converts a string with parameters into an the SInputs array of structures:

void InputsToStruct(string aStr,SInput & aInputs[]){
   string tmp[];
   string tmp2[];
   StringSplit(aStr,',',tmp);
   int sz=ArraySize(tmp);
   ArrayResize(aInputs,sz);
   for(int i=0;i<sz;i++){
      StringSplit(tmp[i],'=',tmp2);
      StringTrimLeft(tmp2[0]);
      StringTrimRight(tmp2[0]);      
      aInputs[i].Name=tmp2[0];
      if(ArraySize(tmp2)>1){
         StringTrimLeft(tmp2[1]);
         StringTrimRight(tmp2[1]);       
         aInputs[i].Value=tmp2[1];
      }
      else{
         aInputs[i].Value="";
      }
   }
}

The string with parameters is divided into an array in accordance with the "," character. Then each element of the resulting array is divided into the name and value based on the character "=".

Now everything is ready for analyzing the extracted data and generating our own reports.

Creating a custom report

Now that we have access to report data, we can analyze it in any preferable way. There are many options. Using trading results, we can calculate general metrics, such as the Sharpe ratio, recovery factor, etc. Custom HTML reports enable access to all HTML, CSS and JavaScript features. Using HTML, we can rearrange reports. For example, we can create a table of orders and deals or separate tables for buy deals and sell deals, and other type of reports

CSS enables data coloring, which makes the report easier to analyze. JavaScript can assist in making the reports interactive. For example, hovering over a deal row can highlight the appropriate order row in the orders table. Images can be generated and added to the HTML report. Optionally, JavaScript allows drawing images directly within the canvas element in html5. It all depends on your needs, imagination and skills.

Let us create a custom html trading report. Let us combine orders and deals in one table. Canceled orders will be shown in gray, because they do not provide information of interest. Also gray will be used for market orders, because they provide the same information as deals executed as a result of the orders. Deals will be highlighted in bright colors: blue and red. Filled pending orders should also have noticeable colors, such as pink and blue. 

The custom report table will contain headers from the order table and the deal table. Let us prepare appropriate arrays:

   string TableHeader[]={  "Time",
                           "Order",
                           "Deal",
                           "Symbol",
                           "Type",
                           "Direction",
                           "Volume",
                           "Price",
                           "Order",
                           "S/L",
                           "T/P",
                           "Time",
                           "State",
                           "Commission",
                           "Swap",
                           "Profit",
                           "Balance",
                           "Comment"};

We receive the structure with the report data:

SHistory History;
HistoryHTMLReportToStruct("ReportHistory-555849.html",History);

Let us open the file for the generated report and write the standard beginning of the HTML page formed by the HTMLStart() function:

   int h=FileOpen("Report.htm",FILE_WRITE);
   if(h==-1)return;

   FileWriteString(h,HTMLStart("Report"));
   FileWriteString(h,"<table>\n");
   
   FileWriteString(h,"\t<tr>\n");   
   for(int i=0;i<ArraySize(TableHeader);i++){
      FileWriteString(h,"\t\t<th>"+TableHeader[i]+"</th>\n"); 
   }
   FileWriteString(h,"\t</tr>\n");     

The HTMLStart() function code is shown below:

string HTMLStart(string aTitle,string aCSSFile="style.css"){
   string str="<!DOCTYPE html>\n";
   str=str+"<html>\n";
   str=str+"<head>\n";
   str=str+"<link href=\""+aCSSFile+"\" rel=\"stylesheet\" type=\"text/css\">\n";
   str=str+"<title>"+aTitle+"</title>\n";
   str=str+"</head>\n";  
   str=str+"<body>\n";     
   return str;
}

A string with the page title for the <title> tag and the file name with styles is passed to the function.

Loop though all orders, define the string display type and form it:

int j=0;
for(int i=0;i<ArraySize(History.Orders);i++){
   
   string sc="";
   
   if(History.Orders[i].State=="canceled"){
      sc="PendingCancelled";
   }
   else if(History.Orders[i].State=="filled"){
      if(History.Orders[i].Type=="buy"){
         sc="OrderMarketBuy";
      }
      else if(History.Orders[i].Type=="sell"){
         sc="OrderMarketSell";
      }
      if(History.Orders[i].Type=="buy limit" || History.Orders[i].Type=="buy stop"){
         sc="OrderPendingBuy";
      }
      else if(History.Orders[i].Type=="sell limit" || History.Orders[i].Type=="sell stop"){
         sc="OrderMarketSell";
      }         
   }

   FileWriteString(h,"\t<tr class=\""+sc+"\">\n");   
   FileWriteString(h,"\t\t<td>"+History.Orders[i].OpenTime+"</td>\n");  // Time
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Order+"</td>\n");     // Order 
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Deal 
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Symbol+"</td>\n");    // Symbol 
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Type+"</td>\n");      // Type 
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Direction       
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Volume+"</td>\n");    // Volume    
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Price+"</td>\n");     // Price  
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Order        
   FileWriteString(h,"\t\t<td>"+History.Orders[i].SL+"</td>\n");        // SL
   FileWriteString(h,"\t\t<td>"+History.Orders[i].TP+"</td>\n");        // TP
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Time+"</td>\n");      // Time    
   FileWriteString(h,"\t\t<td>"+History.Orders[i].State+"</td>\n");     // State
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Comission
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Swap
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Profit     
   FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");                    // Ballance    
   FileWriteString(h,"\t\t<td>"+History.Orders[i].Comment+"</td>\n");   // Comment     
   FileWriteString(h,"\t</tr>\n");   
   

If an order is filled, find an appropriate deal, define its display style and form its HTML code:  

// Find a deal

if(History.Orders[i].State=="filled"){
   for(;j<ArraySize(History.Deals);j++){
      if(History.Deals[j].Order==History.Orders[i].Order){
         
         sc="";
         
         if(History.Deals[j].Type=="buy"){
            sc="DealBuy";
         }
         else if(History.Deals[j].Type=="sell"){
            sc="DealSell";
         }
      
         FileWriteString(h,"\t<tr class=\""+sc+"\">\n");   
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Time+"</td>\n");     // Time
         FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");     // Order 
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Deal+"</td>\n");     // Deal 
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Symbol+"</td>\n");     // Symbol 
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Type+"</td>\n");     // Type 
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Direction+"</td>\n");     // Direction       
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Volume+"</td>\n");     // Volume    
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Price+"</td>\n");     // Price  
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Order+"</td>\n");     // Order        
         FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");     // SL
         FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");     // TP
         FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");     // Time    
         FileWriteString(h,"\t\t<td>"+"&nbsp;"+"</td>\n");     // State
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Commission+"</td>\n");                    // Comission
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Swap+"</td>\n");     // Swap
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Profit+"</td>\n");     // Profit     
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Balance+"</td>\n");     // Ballance    
         FileWriteString(h,"\t\t<td>"+History.Deals[j].Comment+"</td>\n");     // Comment     
         FileWriteString(h,"\t</tr>\n"); 
         break;
      }
   }
}

After that we close the table and add the standard HTML page end, which is formed using the HTMLEnd() function:

FileWriteString(h,"</table>\n");   
FileWriteString(h,HTMLEnd("Report"));  

HTMLEnd() function code:

string HTMLEnd(){
   string str="</body>\n";
   str=str+"</html>\n";
   return str;
}

Now we only need to write the styles file style.css. Learning css is beyond the scope of this article and thus we will not consider this in detail. You can view the file, which is attached below. Also the attachment contains the script creating the report - HTMLReportCreate.mq5.

Here is the ready report:

Fig. 5. Fragment of the custom HTML report

Conclusions

You may wonder if the use of regular expressions would be easier. The overall structure and principle would be the same. We would separately receive an array with the table contents, then rows and cells. Instead of TagsToArray(), we would use a function with the regular expression. The remaining operations would be very similar.

The example of creation of a custom report described in this article is just one of the options for representing reports. It is given only as an example. You can use your own convenient and understandable form. The most important result of the article is that you have easy access to all the report data.

Attachments

• Include/HTMLReport.mqh — include file with the report parsing functions.
• Scripts/HTMLReportTest.mq5 — example of the use of HTMLReport.mqh for parsing testing, optimization and history reports.
• Scripts/HTMLReportCreate.mq5 — example of custom HTML report creation.
• Files/ReportTester-555849.html — Strategy Tester report.
• Files/ReportOptimizer-555849.xml — optimization report.
• Files/ReportHistory-555849.html — trading history report.
• Files/Report.htm — file of the report created using the HTMLReportCreate script.
• Files/style.css — stylesheet for Report.htm.