MQL5: Analysis and Processing of Commodity Futures Trading Commission (CFTC) Reports in MetaTrader 5

Introduction

MetaTrader 5 trading terminal, developed by MetaQuotes Software Corp., as one of the trader's tools contains the MQL5 programming language, that supports object-oriented programming. On one hand, MQL5 is not  compatible with its previous version - MQL4, on the other - it has a lot of new possibilities of fully-featured object-oriented language. So, traders, who already have skills in MQL4, have to comprehend this (in fact) new language, and I consider them as target audience of this article.

In this article, I will show a solution of the following problem: it's necessary to develop the trader tool, that allows to get and analyze different data of Commodity Futures Trading Commission (CFTC) Reports. This topic already has been considered in the article Meta COT Project - New Horizons for CFTC Report Analysis in MetaTrader 4, so I will assume that the reader is familiar with concepts, and we will consider the details that aren't described there.

The idea is following: to develop a single indicator, that allows to get data from files, provided by Commission, without intermediate processing and conversion. Further, it can be used for the different purposes: to plot data as an indicator; as data source in other indicators; in scripts for automated analysis; in the Expert Advisors when developing trading strategies.

1.Types of COT reports

There are three types of reports: Current Legacy Report (COT), Supplemental Commodity Index (CIT),  Current Disaggregated Report (DCOT), the difference between them is presented in Table 1.

Description	Current Legacy Reports	Supplemental Commodity Index	Current Disaggregated Reports
Short name	COT	CIT	DCOT
Refrence title	Current Legacy Reports	Supplemental Commodity Index	Current Disaggregated Reports
Reference to the recent data file	deacot2010.zip	dea_cit_txt_2010.zip	fut_disagg_txt_2010.zip
Groups of traders, presented in the report	Noncommercial Commercial Non-reportable	Noncommercial Commercial Non-reportable Index Traders	Producer/Merchant/Processor/User Swap Dealers Managed Money Other Reportables Non-reportable

Table 1. Types of Commitments of Traders reports  

Information about traders, who should report their positions on the futures markets, principles of classification and periodicity of reports can be found in the "Trade Stocks and Commodities With the Insiders: Secrets of the COT Report" book and in the Meta COT Project - New Horizons for CFTC Report Analysis in MetaTrader 4 article.

I will consider two reports, that aren't described in these sources. 

The CIT acronym stands for Commitments of Index Traders. This report is being published since 2007. The current CIT report web-page is available at  https://cftc.gov/dea/options/deaviewcit.htm. The index traders are in a separate group from the hedgers and speculators.

The description of the Supplemental Report can be found at https://www.cftc.gov/MarketReports/CommitmentsofTraders/AbouttheCOTReports/index.htm. All these index traders often open long positions on the markets and roll-over them from contract to contract.

Since 2009 Commission publishes the Disaggregated Commitments of Traders Report, its description can be found in the explanatory notes.

The Disaggregated COT report increases transparency from the legacy COT reports by separating traders into the following four categories of traders: Producer/Merchant/Processor/User; Swap Dealers; Managed Money; and Other Reportables.

• Producer/Merchant/Processor/User. A “producer/merchant/processor/user” is an entity that predominantly engages in the production, processing, packing or handling of a physical commodity and uses the futures markets to manage or hedge risks associated with those activities.
• Swap Dealer. A “swap dealer” is an entity that deals primarily in swaps for a commodity and uses the futures markets to manage or hedge the risk associated with those swaps transactions. The swap dealer's counterparties may be speculative traders, like hedge funds, or traditional commercial clients that are managing risk arising from their dealings in the physical commodity.
• Money Manager. A “money manager,” for the purpose of this report, is a registered commodity trading advisor (CTA); a registered commodity pool operator (CPO); or an unregistered fund identified by CFTC. These traders are engaged in managing and conducting organized futures trading on behalf of clients.
• Other Reportables. Every other reportable trader that is not placed into one of the other three categories is placed into the “other reportables” category.
  

The full list of reported instruments is presented in the Appendix 2. There are also columns that reflect the presence of particular instrument in the CIT and DCOT reports.

2. Writing a COT indicator, that uses external data from CSV files

The indicator will work as follows. The type of report (one of the listed in Table 1), the type of data and the group of traders are defined in the input parameters of the indicator.

The type of data may be following:

• net longs
  
• net longs rate in open interest
  
• Williams Index, calculated for net longs
  
• open interest
  

The list of possible data types isn't complete, it can be easily extended further to include most essential data, that I use. For the current symbol, the specified data is requested from data files (downloading and unpacking them is described below). We will use the CCFTCReport class for data access and downloading them from these files.

The indicator has the following structure:

2.1. Constants

To define constants the enum data type is used. The report types, supported by indicator, are following:

enum cot_type_report // type of report
{ 
  COT, DCOT, CIT
};

Groups of traders:

enum cot_type_traders // types of traders
{
  noncommercial,      // Non-commercial 
  commercial,         // Commercial
  nonreportable,      // Non-reportable
  index,              // Index traders
  producer,           // Producers
  swapdealers,        // Swap dealers
  managedmoney,       // Managed money
  otherreportables    // Other reportable traders 
};

Data types, that can be used in indicator:

enum cot_type_data    // type of COT data
{   
  netlongs,           // net longs
  netlongsrate,       // net longs rate in the open interest
  willams_index,      // Williams index
  openinterest        // Open interest
}; 

2.2. Classes

To store different objects within a single variable it's possible to use structures and classes. However, it's not possible to assign two variables of structure type, if they contain dynamic arrays or values of a string type. That's why we have used class instead of structure to store COT record, and method for data assignment.

class cot_record                                // class for data record of COT report
  {
public:
   datetime          COT_DATE;                  //date 
   double            COT_OPENINTEREST;          //open interest 
   double            COT_LONGNONCOM;            //longs of non-commercial traders
   double            COT_SHORTNONCOM;           //shorts of non-commercial traders 
   double            COT_LONGCOM;               //longs of commercial traders
   double            COT_SHORTCOM;              //shorts of commercial traders 
   double            COT_LONGNONREPORTABLE;     //longs of the other non-reportable traders
   double            COT_SHORTNONREPORTABLE;    //shorts of the other non-reportable traders
   double            COT_LONGINDEX;             //longs of index traders
   double            COT_SHORTINDEX;            //shorts of index traders
   double            COT_LONGPRODUCER;          //longs of Producer/Merchant/Processor/User
   double            COT_SHORTPRODUCER;         //shorts of Producer/Merchant/Processor/User
   double            COT_LONGSWAPDEALERS;       //longs of Swap Dealers
   double            COT_SHORTSWAPDEALERS;      //shorts of Swap Dealers
   double            COT_LONGMANAGEDMONEY;      //longs of Managed Money traders
   double            COT_SHORTMANAGEDMONEY;     //shorts of the Managed Money traders
   double            COT_LONGOTHERREPORTABLES;  //Other Reportables
   double            COT_SHORTOTHERREPORTABLES;
   string            COT_ID;                    //instrument identifier
   string            COT_NAME;                  //instrument (symbol) name 

   void copyvar(const cot_record &from)         // copying contents (values of variables) from one class to another
     {
      COT_ID                    = from.COT_ID;
      COT_NAME                  = from.COT_NAME;
      COT_DATE                  = from.COT_DATE;
      COT_OPENINTEREST          = from.COT_OPENINTEREST;
      COT_LONGNONCOM            = from.COT_LONGNONCOM;
      COT_SHORTNONCOM           = from.COT_SHORTNONCOM;
      COT_LONGCOM               = from.COT_LONGCOM;
      COT_SHORTCOM              = from.COT_SHORTCOM;
      COT_LONGNONREPORTABLE     = from.COT_LONGNONREPORTABLE;
      COT_SHORTNONREPORTABLE    = from.COT_SHORTNONREPORTABLE;
      COT_LONGINDEX             = from.COT_LONGINDEX;
      COT_SHORTINDEX            = from.COT_SHORTINDEX;
      COT_LONGPRODUCER          = from.COT_LONGPRODUCER;
      COT_SHORTPRODUCER         = from.COT_SHORTPRODUCER;
      COT_LONGSWAPDEALERS       = from.COT_LONGSWAPDEALERS;
      COT_SHORTSWAPDEALERS      = from.COT_SHORTSWAPDEALERS;
      COT_LONGMANAGEDMONEY      = from.COT_LONGMANAGEDMONEY;
      COT_SHORTMANAGEDMONEY     = from.COT_SHORTMANAGEDMONEY;
      COT_LONGOTHERREPORTABLES  = from.COT_LONGOTHERREPORTABLES;
      COT_SHORTOTHERREPORTABLES = from.COT_SHORTOTHERREPORTABLES;
     };
  };

From now I will assume that reader is familiar with CSV format of COT reports, that is considered in the Meta COT Project - New Horizons for CFTC Report Analysis in MetaTrader 4 article. The class instance of this type will be used to store a single line of the COT report. An array of these class instances will allow to use them for storage and convenient access to fields of the COT records. The CCFTCReport class has been developed for data storage and access:

class CCFTCReport                    // COT report
  {
private:
   cot_type_report   type;           //type of current report 
   cot_record        data[];         //cot report data
   int               sizedata;       //number of records in the current report
   string            idsymbol;       //symbol identifier in cot ("CFTC Contract Market Code" field)
   string            terminalsymbol; //symbol name in the client terminal
public:
   void ~CCFTCReport(){ sizedata=0; };  
   bool              Init( cot_type_report passedtype, string sym="" ); //initialization of class instance
   bool              LoadFile( string filename );                       //load data from file
   string            Name();                                            //returns short report name
   bool              Type(cot_type_report passedtype);                  //sets type of report
   cot_type_report Type(){return(type);};                               //returns report type
   bool              TestGroup( cot_type_traders type_trader );         //check for the presence of specified group of traders
   bool              DefineIdSymbol();                                  //definition of id in the cot report
   bool              GetString( int handle, cot_record& arr );          //gets line (record) from csv file   
   string            GetFieldCommaDelimited(string &str);               //gets field from csv string
   double            At(datetime time,cot_type_traders type_trader,cot_type_data typedata=netlongs); //gets data from the report
  };

Instance of CCFTCReport class contains the whole report for a single symbol, the type of report may be COT, CIT or DCOT. The enumerations and classes are listed in the "cot.mqh" included file.

2.3. Input parameters

The input parameters are defined by input variables. These parameters allow to specify group of traders, data type and necessary COT report type:

input cot_type_traders type_trader = noncommercial;  //type of traders
input cot_type_data    type_data   = netlongs;       //type of the indicator
input cot_type_report  type_report = COT;            //type of report

2.4. OnInit function

An indicator has the OnInit function, that is used for the following purposes: to download data from the report file, to check input parameters, to assign indicator's buffers.

The buffer, initialized with the INDICATOR_DATA property, contains data that are plotted on chart. The buffer, initialized with the INDICATOR_CALCULATIONS property, contains intermediate calculations.

SetIndexBuffer( 0, BufferData, INDICATOR_DATA );
IndicatorSetString(INDICATOR_SHORTNAME,indname);
SetIndexBuffer( 1, BufferCalculations, INDICATOR_CALCULATIONS );

2.5. OnCalculate function

This function is used for selection of necessary data, calculation of requested data and their plotting.

Let's consider its work in details. The second form of call is used:

int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &Time[],
                const double &Open[],
                const double &High[],
                const double &Low[],
                const double &Close[],
                const long &TickVolume[],
                const long &Volume[],
                const int &Spread[]){

Let's determine bars for calculation and plotting.

int pos=rates_total-prev_calculated-1;
if( pos >= rates_total-1 )pos--;

Let's specify that elements must be indexed like timeseries.

ArraySetAsSeries(Time, true);
ArraySetAsSeries(BufferData, true);
ArraySetAsSeries(BufferCalculations, true);

After some experiments, I have found that it's better to use timeseries indexation for all arrays, that are related to indicator's buffers. The same is convenient for arrays, that are passed to the OnCalculate function as parameters, the last (recent) element has an index, equal to 0.

The calculation loop is looks as follows: 

for(int i=pos;i>=0;i--)
{
    cot_type_data passed = type_data;
    if(type_data==netlongs || type_data==netlongsrate || type_data==willams_index)
    {
      // For the types of the data (net longs, net longs rate, Williams Index) it returns net longs 
      // Further calculations are based on these data.
      passed = netlongs;
    }
    double res = Report.At(Time[i], type_trader, type_data ); //get report data
    BufferCalculations[i] = res;
    BufferData[i]         = CalcData( type_data, i, Time );
  }

The BufferCalculations array is used to store primary data, selected from the COT report.

The BufferData array contains data ready for plotting on chart. It may be the values from report (net longs, open interest) as well as calculated values (longs rate, Williams Index). I would like to outline the following point, for which I haven't found an elegant solution. Arrays, passed to the OnCalculate function, may be required at subsequent levels, and we can access them only if they are passed in these calls, so I think that it clutters the code.

The CalcData function call:

BufferData[i]         = CalcData( type_data, i, Time );

The CalcData function calls the CalcIndex function:

double CalcData( cot_type_data type_data, int i, const datetime& Time[] )
{
  // net longs rate in the open interest
  if( type_data == netlongsrate ) return( BufferCalculations[ i ] / Report.At( Time[i], type_trader, openinterest ));
  // Williams Index
  if( type_data == willams_index )return( CalcIndex( i, Time, BufferCalculations ) );

  return(BufferCalculations[i]);        
}

I needed access to the Time array inside the CalcIndex function, so I was forced to pass it according the calls hierarchy. Just imagine how the code will look, if we need to use all 8 arrays.

3. Downloading Data Files

All files links are presented in Table 1. There is a separate file for the each year, the year number is present in file name. For example, the https://www.cftc.gov/sites/default/files/files/dea/history/deacot2010.zip file contains report for 2010 year, the https://www.cftc.gov/sites/default/files/files/dea/history/deacot2009.zip file - for 2009, etc.

It's necessary to download all these files and unpack them into \MQL5\files\ folder of Client Terminal install directory. For each year it's necessary to create its own separate folder with a name deacotXXXX, where XXXX corresponds to a year. As a result, we will get the following folder structure.

Fig.1. Folders structure for the reports.

The process of the data preparation can be simplified. All the following operations (checking for the updates on CFTC web-site, downloading and unpacking to the appropriate folders) are performed by "Cotdownloader" script. The script kernel (data downloading) is based on WININET_TEST script. I have used the CProgressBar class, published in the The Price Histogram (Market Profile) and its implementation in MQL5 article. External applications are executed using Windows API, which is described in the Automated Optimization of a Trading Robot in Real Trading article.

Using a script is simple: just attach it to any chart. When working, it reports information about data downloading as a progress bar on the chart and as text messages on the Experts tab.

Fig.2. Data downloading process.

4. Example of Use

To run an indicator, it's necessary to attach it to the chart of chosen symbol and specify the input parameters.

Fig. 3 Input parameters of cot indicator

Note that now you can specify user-friendly representation instead of variable names and values of enumerated data types. It's done as follows: to replace the variable name, you should specify a comment when you declare an input variable:

input cot_type_traders type_trader = noncommercial;  //Type of traders
input cot_type_data    type_data   = netlongs;       //Type of the indicator
input cot_type_report  type_report = COT;            //Type of report

The same is for values - it's necessary to specify the description in comments when declaring the enumerated values:

enum cot_type_data    // types of data
{   
  netlongs,           // Net longs
  netlongsrate,       // Net longs rate in open interest
  willams_index,      // Williams Index
  openinterest        // Open Interest
};

If the input parameters have been specified correctly and the files have been downloaded and unpacked, then indicator will appear in a separate window:

Fig. 4 COT indicator

In the case of errors, they will be printed in "Experts" tab of the "Toolbox" window.

Fig. 5 Error message

5. Release Notes

The indicator isn't positioned as a fully-featured finished product. It's an example that shows how we can get results, that can be used further, by writing of a simple code.  For example, we haven't developed the module, allowing to customize the type of the COT report data with Client Terminal settings, provided by a certain broker. This feature is implemented inside the DefineIdSymbol function. Here are the first few lines of:

bool CCFTCReport::DefineIdSymbol()
  {
   idsymbol="";
   if(terminalsymbol=="USDLFX" || terminalsymbol=="DX_CONT" || StringFind(terminalsymbol,"DX")==0)idsymbol="098662";
   //U.S. DOLLAR INDEX - CE FUTURES U.S.
   if( terminalsymbol == "FC_CONT" || StringFind( terminalsymbol, "FC")== 0)idsymbol = "061641";  //FEEDER CATTLE 
   if( terminalsymbol == "LC_CONT" || StringFind( terminalsymbol, "LC")== 0)idsymbol = "057642";  //LIVE CATTLE 
   if( terminalsymbol == "QM_CONT" || StringFind( terminalsymbol, "QM")== 0)idsymbol = "067651";
   //CRUDE OIL, LIGHT SWEET - NEW YORK MERCANTILE EXCHANGE

If you need to assign a symbol from report to a symbol of client terminal, you should do it manually by modifying this code. I've used settings from BrocoTrader4 terminal. For examples in this article, I've used demo-accounts, opened on Metaquotes-Demo and Alpari-Demo servers. There are only 2 instruments available for DCOT reports: XAUUSD (gold), XAGUSD (silver). 

Currently CIT reports are not supported, because there is no financial instruments on these servers. When they will appear, it's easy to include their support just by uncommenting and modifying some lines in code.

6. How to Use the COT Indicator via iCustom

I don't use indicator in a form, presented at Fig.3. The MQL5 language has the possibility to reuse of the custom indicators using the iCustom function.

6.1. Creating new indicator

There are many ways of COT reports visual analyses. One of them is a histogram, that shows positions of different traders groups with different colors.

Let's consider the following task: to create a COTnet indicator, that plots positions of different traders groups (commercial and non-commercial from COT reports), as a histogram. User must have a possibility to choose the type of data: net longs, net longs rate, Williams Index.

The process can be simplified by MQL5 Wizard, considered in the MQL5: Create Your Own Indicator article. As a result, the code of indicator (drawing styles have been created by MQL5 Wizard) looks as follows:

#property indicator_separate_window
#property indicator_buffers 2
#property indicator_plots   2
//---- plot Noncommercial
#property indicator_label1  "Noncommercial"
#property indicator_type1   DRAW_HISTOGRAM
#property indicator_color1  Blue
#property indicator_style1  STYLE_SOLID
#property indicator_width1  5
//---- plot Commercial
#property indicator_label2  "Commercial"
#property indicator_type2   DRAW_HISTOGRAM
#property indicator_color2  Red
#property indicator_style2  STYLE_SOLID
#property indicator_width2  5
//--- indicator buffers

Input variables, global variables and preprocessor directive to include the library with classes look as follows:

#include <cot.mqh>
input cot_type_data    type_data   = netlongs;  // Indicator's type
double         NoncommercialBuffer[];
double         CommercialBuffer[];
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int handlecomm, handlenoncomm;

The code of the OnInit function is following:

int OnInit()
{
   SetIndexBuffer(0,NoncommercialBuffer,INDICATOR_DATA);
   SetIndexBuffer(1,CommercialBuffer,INDICATOR_DATA);
   cot_type_traders td = commercial;
   cot_type_report  tr = COT; 
   handlecomm = iCustom(NULL, 0, "cot", td, type_data, tr );
   td = noncommercial;
   handlenoncomm = iCustom(NULL, 0, "cot", td, type_data, tr );
   return(0);
}

Let's consider how to use custom indicators (on example of non-commercial traders). After the first call of the iCustom function, the following parameters (necessary for our report) are passed to the function:

• NULL – current symbol in the client terminal
• 0 – current timeframe
• "cot" – file name (note, that it is specified without extension) of our custom indicator
  
•  td = commercial (group of traders that we need)
•  type_data = type of the indicator, it specified in the input parameters
•  tr = COT – type of report
  

As a result, we have a handle, that will be used further to get the data of the indicator.

The getting of the data from the indicator is performed in the function OnCalculate:

  if(BarsCalculated(handlecomm)<rates_total || BarsCalculated(handlenoncomm)<rates_total) return(0);
  int to_copy = clctocopy(rates_total, prev_calculated );
  if( !copycotbuf(handlecomm, CommercialBuffer, to_copy ) )
  {
    Print("Error in data of commercial traders");
    return( 0 );
  }

  if( !copycotbuf(handlenoncomm, NoncommercialBuffer, to_copy ) )
  {
    Print("Error in data of non-commercial traders");
    return( 0 );
  }
  return(rates_total);

We spent some time, wrote a short code and got the result:

Fig. 6 The cotnet indicator

6.2. iCustom can also be used in scripts

Let's consider the following task: creating the cotsignals script to calculate statistical estimations.

Let's check what statistical advantage we'll get, using different data from COT reports. The simplest way is to estimate the probability of correct determination of weekly candle color. Consider the following hypotheses:

• The sign of non-commercial traders' net longs are the same as the candle color: if it's positive, the weekly trend is bullish, if it's negative, the trend is bearish.
• The candle color (white/black) is related directly with increasing/decreasing of non-commercial traders' net longs.
• The candle color is black or white depending on the Williams Index value, calculated on the non-commercial traders' net longs. If it is overbought (more than 75%), then the color is black, if it is oversold (less than 25%) - the candle color is white.
  

We will use data from COT report. Using these examples you can develop your own methods of COT data interpretation and get advantage of it. In your case we will check for 150 weekly bars of history, that corresponds to a period of 3 years. Before writing a script, we have to define the necessary data types and classes:

Enumeration for possible types of statistics is:

enum cot_type_statistics   //types of statistics, used for calculation 
{  
  speculantsign,           //Sign of non-commercial traders' net longs 
  speculantchange,         //Volume changes of non-commercial traders' longs 
  willamszones             //Overbought/oversold zones of Williams Index
};

To get the statistics for the specified symbol, let's create special class:

class CCOTStatistic
{
  private:
    
    int bars ;                            // number of weekly bars in symbol history
    string symbol;                        // symbol name in Client Terminal
    double netlongsspeculantsdata[];      // data for speculators' net longs 
    double willamsindexspeculantsdata[];  // data for speculators' Williams index net longs   
    cot_type_statistics liststatistic[];  // list of statistics
  public:
    //initialization and initial setup
    void Init( string symbol_passed, cot_type_statistics& listpassed[] );
    //Get the forecast for the direction 
    int GetCotSignalDirection( int ibar, cot_type_statistics type_stat   );
    bool Get( double& probably[] );       // returns statistics
};

The requested statistics are returned by the Get method, let's describe how it works. First, the necessary arrays are filled with data using custom indicator. We need non-commercial (speculators) traders' net longs values and Williams Index values:

if (!LoadDataFromCOTIndicator(symbol, PERIOD_W1, noncommercial, netlongs, COT, netlongsspeculantsdata, bars))
   return(false);
  // we have got the data - speculators' net longs
if (!LoadDataFromCOTIndicator(symbol, PERIOD_W1, noncommercial, willams_index, COT, willamsindexspeculantsdata, bars))
   return(false);
  // we have got the data - speculators' Williams Index

The code for statistics calculation (we have only one parameter to be calculated - it is a forecast probability for the color of weekly candle) looks as follows:

for(int istat = 0; istat < ArraySize(liststatistic); istat++)
  {
   int cntsignals = 0;
   int cntsucsess = 0;
   for(int i=bars-1; i>=0; i--)
     {
      double colorcandle=iClose(symbol,PERIOD_W1,i)-iOpen(symbol,PERIOD_W1,i);
      if(symbol=="USDCHF" || symbol=="USDCAD" || symbol=="USDJPY") colorcandle=-colorcandle;
      double cotdirection=GetCotSignalDirection(i,liststatistic[istat]);
      if(cotdirection==0)continue;                 //no signal
      cntsignals++;
      if(cotdirection*colorcandle>0) cntsucsess++; //color and direction are same
     }

   if(cntsignals!=0) probably[istat]=1.*cntsucsess/cntsignals;

   Print("Calculation complete for ",symbol,GetStringTypeStatistic(liststatistic[istat]),
                 " the correct forecast probability=",DoubleToString(probably[istat],2));
   Print("Total signals:",cntsignals,"success:",cntsucsess);
  }

The LoadDataFromCOTIndicator function is complicated, because it's necessary to perform numerous of checks to get data from indicator, so let's consider it in detail:

bool LoadDataFromCOTIndicator( string           symbol,        //symbol name
                               ENUM_TIMEFRAMES  timeframe,     //timeframe
                               cot_type_traders type_trader,   //group of traders type
                               cot_type_data    type_data  ,   //indicator type 
                               cot_type_report  type_report,   //report type
                               double&          loadto[],      //output data array
                               int              nbars   )      //number of requested bars 
{     
  
  int cothandle;
  nbars+=10 ; //for reserve
  cothandle = iCustom( symbol , timeframe, "cot", type_trader, type_data, type_report );

In the last line we get the handle of custom indicator, that can be used to get data from buffers.

We have to check that the indicator has been created successfully:

  if( cothandle == INVALID_HANDLE ){
    Print("Error in indicator creation for symbol ", symbol );
    return(false);
  }

Checking for history data, that we need:

int n = BarsSinh( symbol, timeframe );
  if(  n < nbars    )
{ 

If there is not enough history data, let's load it:

Print("Loading history for ", symbol ); //, " start at ", loadtm  
    CheckLoadHistory( symbol ,timeframe,   loadtm );
    n = BarsSinh( symbol, timeframe );
    if( n < nbars )
    {
      Print("Not enough history for the "+symbol, " total ", n, "bars");
      return(false);
    }
  }

The BarsSinh (returns the number of bars, available on the specified symbol and timeframe) and CheckLoadHistory (downloading of the history) function are based on the script code from the MQL5 Reference.

Before using data from the indicator's buffer, we have to check, that data has been prepared, because just the presence of a handle doesn't guarantee that calculation has been finished.

The next call performs this check and waits when calculations are finished:

if( !WaitForCalculcateIndicator( cothandle ) ){
    Print("Timeout for the COT indicator calculation");
    return(false);
  }

The data are ready, we have to copy them:

int res = CopyBuffer(cothandle, 0, 0, nbars, loadto );

Next we're checking, that data has been copied successfully:

if( res != nbars ){
    Print("Error in obtaining the data, ", MyErrorDescription(_LastError) );
    return(false);
  }

And finally:

return(true);

We've got the data and return them into array, passed by function parameter.

The resulted statistics will be printed into CSV file, the CCOTOutFile class is developed for this purpose:

class CCOTOutFile
{
  private:
   string  filename;
   int     handle; 
  public:
    bool Init( string passedname );
    void PutString( string symbol, double& probably[] );
    void PutHeader( cot_type_statistics& arrheaders[] ) ;
    void ~CCOTOutFile(){       FileClose(handle);    }
};

It creates the file with output data, writes strings in .csv format, forms the header and closes file while destruction of class instance.

The code of the script will be short, because all necessary libraries have been written.

The Input parameters and included libraries:

input int      NBars =150; //History depth W1
#include <common.mqh>
#include <cot.mqh>

The list of symbols for analysis:

string listsymbols[] = {"EURUSD", "GBPUSD", "USDCHF", "USDCAD", "USDJPY", "AUDUSD", "XAUUSD"};

The object initialization:

void OnStart()
{
  CCOTOutFile outfile;  //file with results
  if( !outfile.Init("cotw1signals.csv") )return;

The list of statistics for calculation:

  cot_type_statistics list_statistic[]= // statistic types list
  { speculantsign, speculantchange, willamszones };

Writing CSV file header:

outfile.PutHeader( list_statistic ); //writing CSV file header

In the main loop we get statistics for each symbol and signal, and write results to file:

  for( int i = 0; i < ArraySize(listsymbols); i++  ) //for all symbols in the list
 {  
    Print( "Analysis for "+listsymbols[i] );
    CCOTStatistic stat;
    stat.Init( listsymbols[i ], list_statistic );
    double probably[];                               //probability of a signal
    if( !stat.Get( probably ) )
    {
      Print( "Error in statistics calculation for the symbol ", listsymbols[i] );
      return;
    }  
    outfile.PutString( listsymbols[i], probably );   //write string to .csv file
  }
  Print("Calculation complete.");
}

The script has been tested on Alpari-Demo server. If you will run it on MetaQuotes-Demo server, it will print the message "Error in statistics calculation for the XAUUSD symbol", because this symbol is not available now.

As a result of script execution we will get the file, that looks as follows:

To make it more clear, let's open it Excel, calculate the average values and create a histogram of probabilities:

Fig. 8 Forecast probability

The forecast results are the same for all symbols. The averaged values of correct forecast probability for different types of signals are:

• 0.54 – non-commercial traders sign;
• 0.50 – volume changes in net longs of non-commercial traders;
• 0.48 – overbought/oversold zones of Williams Index.

As we see, we have got the best forecast results for the non-commercial traders' net longs. The worst result is for the zones of Williams Index. The value 0.48 means that the probability for white candle is equal to 0.52, even if Williams Index is overbought, and black color for the case, if it is oversold. So, its use in the form, presented by Williams, isn't rational. Maybe, the results can be improved using larger timeframes: month, or maybe larger. We have used all the symbols that are available on the demo servers and in the COT reports.

Conclusion

MQL5 is a tool, that allows you to program the whole cycle of tasks, necessary to develop a trading system:

• The indicators with complicated algorithms, that are used to access external data sources;
• Its reuse in other indicators, scripts and Expert Advisors;
• The implementation of own algorithms of statistical and quantitative analysis.

Pros:

• Object-oriented programming greatly reduces time spent for debugging.
  
• Debugger.
  
• It's easy to migrate from MQL4 to MQL5.
  
• Implementation of object-oriented model is successful, it's easy to use it. 
  
• Documentation is well organized. 
  
• Integration with Windows API extends the platform, for example it allows to work with Internet pages.

Cons:

The main trouble for me was the access to historical data:  

• Lack of necessary primitive functions to access timeseries (as Time[], Open[], Close[]  and others in MQL4)
• When accessing data, it's necessary to perform a numerous checks of their accessibility, it's necessary to understand their details.
  

There is a debugger, but it hasn't many useful features: there no indicators debugging, also it's impossible to perform checks of complicated objects, like arrays and classes. The list of the pros and cons isn't comprehensive, I have listed only what I've came across while preparing this article.

Appendix 1. List of files

All files are located in Client Terminal folder. Unpack the files from sources.zip to Client Terminal folder.

№	File name	Description
1	MQL5\Files\unzip.exe	Windows application for unpacking of .zip archives
2	MQL5\Include\Cdownloader.mqh	Class, that downloads CFTC archives from the Internet
3	MQL5\Include\ClassProgressBar.mqh	Class CProgressBar, that is used to show the downloading process in the chart window
4	MQL5\Include\common.mqh	Common functions and constants, that used in all indicators and scripts
5	MQL5\Include\cot.mqh	Class CCFTCReport, that selects the data from cot reports
6	MQL5\Include\ErrorDescription.mqh	Error Description library
7	MQL5\Indicators\cot.mq5	The base indicator cot
8	MQL5\Indicators\cotnet.mq5	The indicator cotnet, a simple example of use of cot.mq5 as custom user indicator
9	MQL5\Scripts\cotdownloader.mq5	Script cotdownloader, it downloads the archive files from the Internet
10	MQL5\Scripts\cotsignals.mq5	Script cotsignals, example of statistical analysis of COT reports

Table 2. List of files

Appendix 2. List of symbols, available in COT reports

№	Symbol name	ID on exchange	ID in Client Terminal	Present in CIT	Present in DCOT
1	WHEAT - CHICAGO BOARD OF TRADE	001602	ZW	x	x
2	WHEAT - KANSAS CITY BOARD OF TRADE	001612		x	x
3	WHEAT - MINNEAPOLIS GRAIN EXCHANGE	001626			x
4	CORN - CHICAGO BOARD OF TRADE	002602	ZC	x	x
5	OATS - CHICAGO BOARD OF TRADE	004603	ZO		x
6	SOYBEANS - CHICAGO BOARD OF TRADE	005602	ZS	x	x
7	MINI SOYBEANS - CHICAGO BOARD OF TRADE	005603			x
8	SULFUR FINANCIAL INSTRUMENT - CHICAGO CLIMATE FUTURES EXCHANGE	006261			x
9	CARBON FINANCIAL INSTRUMENT - CHICAGO CLIMATE FUTURES EXCHANGE	006268			x
10	RGGI CO2 ALLOWANCE 2009 - CHICAGO CLIMATE FUTURES EXCHANGE	00626U			x
11	SOYBEAN OIL - CHICAGO BOARD OF TRADE	007601	ZL	x	x
12	U.S. TREASURY BONDS - CHICAGO BOARD OF TRADE	020601	ZB
13	LONG-TERM U.S. TREASURY BONDS - CHICAGO BOARD OF TRADE	020604
14	GULF # 6 FUEL 3.0% SULFUR SWAP - NEW YORK MERCANTILE EXCHANGE	02165A			x
15	NY RES FUEL 1.0% SULFUR SWAP - NEW YORK MERCANTILE EXCHANGE	02165B			x
16	EUR 1% FUEL OIL NWE CAL SWAP - NEW YORK MERCANTILE EXCHANGE	02165C			x
17	EUR 3.5% FUEL OIL RTD CAL SWAP - NEW YORK MERCANTILE EXCHANGE	02165E			x
18	SING FUEL OIL 180 CAL SWAP - NEW YORK MERCANTILE EXCHANGE	02165G			x
19	EAST WEST FUEL OIL SPR SWAP - NEW YORK MERCANTILE EXCHANGE	02165I			x
20	NY 1% V GULF 3% FUEL OIL SPR - NEW YORK MERCANTILE EXCHANGE	02165T			x
21	NO. 2 HEATING OIL	022651			x
22	SING GASOIL SWAP - NEW YORK MERCANTILE EXCHANGE	02265J			x
23	SING GASOIL/RDAM GASOIL SWAP - NEW YORK MERCANTILE EXCHANGE	02265T			x
24	NYMEX HEATING OIL/RDAM GASOIL - NEW YORK MERCANTILE EXCHANGE	02265U			x
25	GASOIL (ICE) SWAP - NEW YORK MERCANTILE EXCHANGE	02265V			x
26	UP DOWN GC ULSD VS HO SPR SWAP - NEW YORK MERCANTILE EXCHANGE	022A13			x
27	SING GASOIL BALMO SWAP - NEW YORK MERCANTILE EXCHANGE	022A22			x
28	NATURAL GAS ICE HENRY HUB - ICE OTC	023391			x
29	NATURAL GAS - NEW YORK MERCANTILE EXCHANGE	023651	QG		x
30	MICHCON BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365A			x
31	M-3 BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365C			x
32	TCO BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365D
33	NGPL TEXOK BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365G			x
34	NGPL MIDCON BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365K			x
35	WAHA BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	02365O			x
36	HOUSTON SHIP CH INDEX SWAP - NEW YORK MERCANTILE EXCHANGE	023A10			x
37	CBT ETHANOL - CHICAGO BOARD OF TRADE	025601			x
38	CHICAGO ETHANOL SWAP - NEW YORK MERCANTILE EXCHANGE	025651			x
39	SOYBEAN MEAL - CHICAGO BOARD OF TRADE	026603	ZM		x
40	JAPAN C&F NAPTHA SWAP - NEW YORK MERCANTILE EXCHANGE	03265C			x
41	COTTON NO. 2 - ICE FUTURES U.S.	033661	CT	x	x
42	HENRY HUB BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	035652			x
43	HOUSTON SHIP CH BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	035653			x
44	NW PIPE ROCKIES BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	035654			x
45	PANHANDLE BASIS SWAP - NEW YORK MERCANTILE EXCHANGE	035655			x
46	HENRY HUB SWAP - NEW YORK MERCANTILE EXCHANGE	03565B
47	HENRY HUB PENULTIMATE GAS SWAP - NEW YORK MERCANTILE EXCHANGE	03565C			x
48	ROUGH RICE - CHICAGO BOARD OF TRADE	039601	ZR		x
49	FRZN CONCENTRATED ORANGE JUICE - ICE FUTURES U.S.	040701	JO		x
50	2-YEAR U.S. TREASURY NOTES - CHICAGO BOARD OF TRADE	042601
51	10-YEAR U.S. TREASURY NOTES - CHICAGO BOARD OF TRADE	043602	ZN
52	5-YEAR U.S. TREASURY NOTES - CHICAGO BOARD OF TRADE	044601
53	30-DAY FEDERAL FUNDS - CHICAGO BOARD OF TRADE	045601	ZQ
54	MILK Class III - CHICAGO MERCANTILE EXCHANGE	052641			x
55	LEAN HOGS - CHICAGO MERCANTILE EXCHANGE	054642	HE	x	x
56	LIVE CATTLE - CHICAGO MERCANTILE EXCHANGE	057642	LC	x	x
57	RANDOM LENGTH LUMBER - CHICAGO MERCANTILE EXCHANGE	058643	LB		x
58	FEEDER CATTLE - CHICAGO MERCANTILE EXCHANGE	061641	FC	x	x
59	PJM ELECTRICITY MONTHLY - NEW YORK MERCANTILE EXCHANGE	064657			x
60	ISO NEW ENGLAND LMP SWAP - NEW YORK MERCANTILE EXCHANGE	06465H			x
61	PJM CAL MONTH OFF PK LMP SWAP - NEW YORK MERCANTILE EXCHANGE	06465M			x
62	ISO NEW ENG OFF PK LMP SWAP - NEW YORK MERCANTILE EXCHANGE	06465S			x
63	CINERGY CAL MONTH LMP SWAP - NEW YORK MERCANTILE EXCHANGE	064A01			x
64	CINERGY OFF PEAK LMP SWAP - NEW YORK MERCANTILE EXCHANGE	064A02
65	PJM N ILL PEAK DAY AHEAD - NEW YORK MERCANTILE EXCHANGE	064A34			x
66	PJM JCPL PEAK DAY AHEAD - NEW YORK MERCANTILE EXCHANGE	064A48			x
67	PJM PEPCO PEAK DAY AHEAD - NEW YORK MERCANTILE EXCHANGE	064A50			x
68	PJM PSEG PEAK DAY AHEAD - NEW YORK MERCANTILE EXCHANGE	064A54			x
69	PJM WESTERN PEAK DAY AHEAD - NEW YORK MERCANTILE EXCHANGE	064A56
70	PJM WESTERN PEAK REAL TIME - NEW YORK MERCANTILE EXCHANGE	064A58			x
71	PJM WESTERN OFF PEAK REAL TIME - NEW YORK MERCANTILE EXCHANGE	064A59			x
72	ISO NEW ENG INT HUB PEAK SWAP - NEW YORK MERCANTILE EXCHANGE	064A60			x
73	MW IND TRANS PEAK SWAP - NEW YORK MERCANTILE EXCHANGE	064A62			x
74	NYISO ZONE 5 MW PEAK SWAP - NEW YORK MERCANTILE EXCHANGE	064A66
75	ISO NEW ENG HUB OFF PEAK SWAP - NEW YORK MERCANTILE EXCHANGE	064A78
76	MT BELVIEU PROPANE 5 DEC SWAP - NEW YORK MERCANTILE EXCHANGE	06665O
77	MT BELVIEU ETHANE 5 DEC SWAP - NEW YORK MERCANTILE EXCHANGE	06665P			x
78	MT BELV NORM BUTANE 5 DEC SWAP - NEW YORK MERCANTILE EXCHANGE	06665Q			x
79	MT BELV NAT GASOLINE 5 DEC SWP - NEW YORK MERCANTILE EXCHANGE	06665R			x
80	CRUDE OIL LIGHT SWEET - ICE FUTURES EUROPE   LIGHT SWEET - ICE FUTURES EUROPE	067411			x
81	CRUDE OIL, LIGHT SWEET - NEW YORK MERCANTILE EXCHANGE	067651	QM		x
82	WTI CRUDE OIL CALENDAR SWAP - NEW YORK MERCANTILE EXCHANGE	06765A			x
83	DUBAI CRUDE OIL CALENDAR SWAP - NEW YORK MERCANTILE EXCHANGE	06765G			x
84	WTI CRUDE OIL FINANCIAL - NEW YORK MERCANTILE EXCHANGE	06765I			x
85	BRENT FINANCIAL - NEW YORK MERCANTILE EXCHANGE	06765J			x
86	BRENT (ICE) CALENDAR SWAP - NEW YORK MERCANTILE EXCHANGE	06765N			x
87	BRENT-DUBAI SWAP - NEW YORK MERCANTILE EXCHANGE	06765O			x
88	COCOA - ICE FUTURES U.S.	073732	CC	x	x
89	PALLADIUM - NEW YORK MERCANTILE EXCHANGE	075651	PA		x
90	PLATINUM - NEW YORK MERCANTILE EXCHANGE	076651	PL		x
91	SUGAR NO. 11 - ICE FUTURES U.S.	080732	SB	x	x
92	COFFEE C - ICE FUTURES U.S.	083731	KC	x	x
93	SILVER - COMMODITY EXCHANGE INC.	084691	SI,XAGUSD,ZI		x
94	COPPER-GRADE #1 - COMMODITY EXCHANGE INC.	085692	HG		x
95	GOLD - COMMODITY EXCHANGE INC.	088691	GC,GOLD,XAUUSD		x
96	RUSSIAN RUBLE - CHICAGO MERCANTILE EXCHANGE	089741	USDRUB,USDRUR
97	CANADIAN DOLLAR - CHICAGO MERCANTILE EXCHANGE	090741	6C,USDCAD
98	SWISS FRANC - CHICAGO MERCANTILE EXCHANGE	092741	6S,USDCHF
99	MEXICAN PESO - CHICAGO MERCANTILE EXCHANGE	095741
100	BRITISH POUND STERLING - CHICAGO MERCANTILE EXCHANGE	096742	6B,GBPUSD
101	JAPANESE YEN - CHICAGO MERCANTILE EXCHANGE	097741	6J,USDJPY
102	U.S. DOLLAR INDEX - ICE FUTURES U.S.	098662	DX
103	EURO FX - CHICAGO MERCANTILE EXCHANGE	099741	6E,EURUSD
104	GASOLINE BLENDSTOCK (RBOB) - NEW YORK MERCANTILE EXCHANGE	111659	XRB		x
105	RBOB CALENDAR SWAP - NEW YORK MERCANTILE EXCHANGE	11165K			x
106	NEW ZEALAND DOLLAR - CHICAGO MERCANTILE EXCHANGE	112741	6N,NZDUSD
107	VIX FUTURES - CBOE FUTURES EXCHANGE	011700
108	DOW JONES INDUSTRIAL AVERAGE - CHICAGO BOARD OF TRADE	124601
109	3-MONTH EURODOLLARS - CHICAGO MERCANTILE EXCHANGE	132741
110	S&P 500 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE	138741
111	E-MINI S&P 500 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE	13874A	ES,SPX
112	NASDAQ-100 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE	209741	NQ
113	NASDAQ-100 STOCK INDEX (MINI) - CHICAGO MERCANTILE EXCHANGE	209742
114	DOW JONES UBS EXCESS RETURN - CHICAGO BOARD OF TRADE	221602
115	AUSTRALIAN DOLLAR - CHICAGO MERCANTILE EXCHANGE	232741	6A,AUDUSD
116	RUSSELL 2000 MINI INDEX FUTURE - ICE FUTURES U.S.	23977A
117	NIKKEI STOCK AVERAGE - CHICAGO MERCANTILE EXCHANGE	240741
118	NIKKEI STOCK AVERAGE YEN DENOM - CHICAGO MERCANTILE EXCHANGE	240743
119	E-MINI MSCI EAFE - CHICAGO MERCANTILE EXCHANGE	244741
120	E-MINI MSCI EMERGING MARKETS - CHICAGO MERCANTILE EXCHANGE	244742
121	INTEREST RATE SWAPS 10YR - CHICAGO BOARD OF TRADE	246602
122	INTEREST RATE SWAPS 5YR - CHICAGO BOARD OF TRADE	247602
123	S&P GSCI COMMODITY INDEX - CHICAGO MERCANTILE EXCHANGE	256741
124	SING JET KERO SWAP - NEW YORK MERCANTILE EXCHANGE	26265D
125	E-MINI S&P 400 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE	33874A
126	GULF JET NY HEAT OIL SPR SWAP - NEW YORK MERCANTILE EXCHANGE	86465A			x
127	SING JET KERO GASOIL SPR SWAP - NEW YORK MERCANTILE EXCHANGE	86465C			x
128	JET CIF NWE/GASOIL FUT - NEW YORK MERCANTILE EXCHANGE	86465D			x
129	GULF # 6 FUEL OIL CRACK SWAP - NEW YORK MERCANTILE EXCHANGE	86565A			x
130	3.5% FUEL OIL RDAM CRACK SPR - NEW YORK MERCANTILE EXCHANGE	86565C			x
131	NAPTHA CRACK SPR SWAP - NEW YORK MERCANTILE EXCHANGE	86665A			x
132	GASOIL CRACK SPR SWAP - NEW YORK MERCANTILE EXCHANGE	86765C			x

Table 3. List of symbols, available in COT reports