MetaTrader 5 / Examples
Русский 中文 Español Deutsch 日本語 Português 한국어 Français Italiano Türkçe
preview
Evaluating ONNX models using regression metrics

Evaluating ONNX models using regression metrics

MetaTrader 5 — Examples | 19 September 2023, 14:40
8 114 0
MetaQuotes
MetaQuotes

Introduction

Regression is a task of predicting a real value from an unlabeled example. A well-known example of regression is estimating the value of a diamond based on such characteristics as size, weight, color, clarity, etc.

The so-called regression metrics are used to assess the accuracy of regression model predictions. Despite similar algorithms, regression metrics are semantically different from similar loss functions. It is important to understand the difference between them. It can be formulated as follows:

  • The loss function arises at the moment when we reduce the problem of building a model to an optimization problem. It is usually required that it has good properties (e.g. differentiability).

  • A metric is an external objective quality criterion, usually depending not on the model parameters, but only on the predicted values.


Regression metrics in MQL5

MQL5 language features the following metrics:

  • Mean Absolute Error, MAE
  • Mean Squared Error, MSE
  • Root Mean Squared Error, RMSE
  • R-squared, R2
  • Mean Absolute Percentage Error, MAPE
  • Mean Squared Percentage Error, MSPE
  • Root Mean Squared Logarithmic Error, RMSLE

It is expected that the number of regression metrics in MQL5 will be increased.


Brief characteristics of regression metrics

MAE estimates the absolute error - how much the predicted number diverged from the actual number. The error is measured in the same units as the value of the objective function. The error value is interpreted based on the range of possible values. For example, if the target values are in the range from 1 to 1.5, then the average absolute error with a value of 10 is a very large error; for the range of 10000...15000 it is quite acceptable. It is not suitable for evaluating forecasts with a large spread of values.

In MSE, each error has its own weight due to squaring. Large discrepancies between the forecast and reality are much more noticeable because of this.

RMSE has the same advantages as MSE, but is more convenient for understanding, since the error is measured in the same units as the values of the objective function. It is very sensitive to anomalies and spikes. MAE and RMSE can be used together to determine error variation in a set of predictions. RMSE is always greater than or equal to MAE. The greater the difference between them, the greater the spread of individual errors in the sample. If RMSE = MAE, all errors have the same magnitude.

R2 — determination ratio shows the strength of the relationship between two random variables. It helps to determine the share of data diversity the model was able to explain. If the model always predicts accurately, the metric is 1. For the trivial model, it is 0. The metric value can be negative if the model predicts worse than the trivial one while the model does not follow the data trend.

MAPE - the error has no dimension and is very easy to interpret. It can be expressed both as decimals and as percentage. In MQL5 it is expressed in decimals. For example, a value of 0.1 indicates that the error was 10% of the actual value. The idea behind this metric is sensitivity to relative deviations. It is not suitable for tasks where you need to work with real units of measurement.

MSPE can be considered as a weighted version of MSE, where the weight is inversely proportional to the square of the observed value. Thus, as the observed values increase, the error tends to decrease.

RMSLE is used when actual values extend over several orders of magnitude. By definition, predicted and actual observed values cannot be negative.

The algorithms for calculating all the above metrics are provided in the source file VectorRegressionMetric.mqh


ONNX models

We used 4 regression models predicting the closing price of the day (EURUSD, D1) from the previous daily bars. We considered these models in the previous articles: "Wrapping ONNX models in classes", "An example of how to ensemble ONNX models in MQL5" and "How to use ONNX models in MQL5". Therefore, we will not repeat here the rules used to train the models. The scripts for training all models are located in the Python subfolder of the zip archive attached to this article. Trained onnx models — model.eurusd.D1.10, model.eurusd.D1.30, model.eurusd.D1.52 and model.eurusd.D1.63 are located there as well.


Wrapping ONNX models in classes

In the previous article, we showed the base class for ONNX models and derived classes for classification models. We have implemented some minor changes to the base class to make it more flexible. 

//+------------------------------------------------------------------+
//|                                            ModelSymbolPeriod.mqh |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+

//--- price movement prediction
#define PRICE_UP   0
#define PRICE_SAME 1
#define PRICE_DOWN 2

//+------------------------------------------------------------------+
//| Base class for models based on trained symbol and period         |
//+------------------------------------------------------------------+
class CModelSymbolPeriod
  {
protected:
   string            m_name;             // model name
   long              m_handle;           // created model session handle
   string            m_symbol;           // symbol of trained data
   ENUM_TIMEFRAMES   m_period;           // timeframe of trained data
   datetime          m_next_bar;         // time of next bar (we work at bar begin only)
   double            m_class_delta;      // delta to recognize "price the same" in regression models

public:
   //+------------------------------------------------------------------+
   //| Constructor                                                      |
   //+------------------------------------------------------------------+
   CModelSymbolPeriod(const string symbol,const ENUM_TIMEFRAMES period,const double class_delta=0.0001)
     {
      m_name="";
      m_handle=INVALID_HANDLE;
      m_symbol=symbol;
      m_period=period;
      m_next_bar=0;
      m_class_delta=class_delta;
     }

   //+------------------------------------------------------------------+
   //| Destructor                                                       |
   //+------------------------------------------------------------------+
   ~CModelSymbolPeriod(void)
     {
      Shutdown();
     }

   //+------------------------------------------------------------------+
   //|                                                                  |
   //+------------------------------------------------------------------+
   string GetModelName(void)
     {
      return(m_name);
     }

   //+------------------------------------------------------------------+
   //| virtual stub for Init                                            |
   //+------------------------------------------------------------------+
   virtual bool Init(const string symbol, const ENUM_TIMEFRAMES period)
     {
      return(false);
     }

   //+------------------------------------------------------------------+
   //| Check for initialization, create model                           |
   //+------------------------------------------------------------------+
   bool CheckInit(const string symbol, const ENUM_TIMEFRAMES period,const uchar& model[])
     {
      //--- check symbol, period
      if(symbol!=m_symbol || period!=m_period)
        {
         PrintFormat("Model must work with %s,%s",m_symbol,EnumToString(m_period));
         return(false);
        }

      //--- create a model from static buffer
      m_handle=OnnxCreateFromBuffer(model,ONNX_DEFAULT);
      if(m_handle==INVALID_HANDLE)
        {
         Print("OnnxCreateFromBuffer error ",GetLastError());
         return(false);
        }

      //--- ok
      return(true);
     }

   //+------------------------------------------------------------------+
   //| Release ONNX session                                             |
   //+------------------------------------------------------------------+
   void Shutdown(void)
     {
      if(m_handle!=INVALID_HANDLE)
        {
         OnnxRelease(m_handle);
         m_handle=INVALID_HANDLE;
        }
     }

   //+------------------------------------------------------------------+
   //| Check for continue OnTick                                        |
   //+------------------------------------------------------------------+
   virtual bool CheckOnTick(void)
     {
      //--- check new bar
      if(TimeCurrent()<m_next_bar)
         return(false);
      //--- set next bar time
      m_next_bar=TimeCurrent();
      m_next_bar-=m_next_bar%PeriodSeconds(m_period);
      m_next_bar+=PeriodSeconds(m_period);

      //--- work on new day bar
      return(true);
     }

   //+------------------------------------------------------------------+
   //| virtual stub for PredictPrice (regression model)                 |
   //+------------------------------------------------------------------+
   virtual double PredictPrice(datetime date)
     {
      return(DBL_MAX);
     }

   //+------------------------------------------------------------------+
   //| Predict class (regression ~> classification)                     |
   //+------------------------------------------------------------------+
   virtual int PredictClass(datetime date,vector& probabilities)
     {
      date-=date%PeriodSeconds(m_period);
      double predicted_price=PredictPrice(date);
      if(predicted_price==DBL_MAX)
         return(-1);

      double last_close[2];
      if(CopyClose(m_symbol,m_period,date,2,last_close)!=2)
         return(-1);
      double prev_price=last_close[0];

      //--- classify predicted price movement
      int    predicted_class=-1;
      double delta=prev_price-predicted_price;
      if(fabs(delta)<=m_class_delta)
         predicted_class=PRICE_SAME;
      else
        {
         if(delta<0)
            predicted_class=PRICE_UP;
         else
            predicted_class=PRICE_DOWN;
        }

      //--- set predicted probability as 1.0
      probabilities.Fill(0);
      if(predicted_class<(int)probabilities.Size())
         probabilities[predicted_class]=1;
      //--- and return predicted class
      return(predicted_class);
     }
  };
//+------------------------------------------------------------------+

We have added a datetime parameter to the PredictPrice and PredictClass methods so that we can make predictions for any point in time, not just for the current one. This will be useful for forming a prediction vector.


D1_10 model class

Our first model is called model.eurusd.D1.10.onnx. Regression model trained on EURUSD D1 on the series of 10 OHLC prices. 
//+------------------------------------------------------------------+
//|                                             ModelEurusdD1_10.mqh |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#include "ModelSymbolPeriod.mqh"

#resource "Python/model.eurusd.D1.10.onnx" as uchar model_eurusd_D1_10[]

//+------------------------------------------------------------------+
//| ONNX-model wrapper class                                         |
//+------------------------------------------------------------------+
class CModelEurusdD1_10 : public CModelSymbolPeriod
  {
private:
   int               m_sample_size;

public:
   //+------------------------------------------------------------------+
   //| Constructor                                                      |
   //+------------------------------------------------------------------+
   CModelEurusdD1_10(void) : CModelSymbolPeriod("EURUSD",PERIOD_D1)
     {
      m_name="D1_10";
      m_sample_size=10;
     }

   //+------------------------------------------------------------------+
   //| ONNX-model initialization                                        |
   //+------------------------------------------------------------------+
   virtual bool Init(const string symbol, const ENUM_TIMEFRAMES period)
     {
      //--- check symbol, period, create model
      if(!CModelSymbolPeriod::CheckInit(symbol,period,model_eurusd_D1_10))
        {
         Print("model_eurusd_D1_10 : initialization error");
         return(false);
        }

      //--- since not all sizes defined in the input tensor we must set them explicitly
      //--- first index - batch size, second index - series size, third index - number of series (OHLC)
      const long input_shape[] = {1,m_sample_size,4};
      if(!OnnxSetInputShape(m_handle,0,input_shape))
        {
         Print("model_eurusd_D1_10 : OnnxSetInputShape error ",GetLastError());
         return(false);
        }
   
      //--- since not all sizes defined in the output tensor we must set them explicitly
      //--- first index - batch size, must match the batch size of the input tensor
      //--- second index - number of predicted prices
      const long output_shape[] = {1,1};
      if(!OnnxSetOutputShape(m_handle,0,output_shape))
        {
         Print("model_eurusd_D1_10 : OnnxSetOutputShape error ",GetLastError());
         return(false);
        }
      //--- ok
      return(true);
     }

   //+------------------------------------------------------------------+
   //| Predict price                                                    |
   //+------------------------------------------------------------------+
   virtual double PredictPrice(datetime date)
     {
      static matrixf input_data(m_sample_size,4);    // matrix for prepared input data
      static vectorf output_data(1);                 // vector to get result
      static matrix  mm(m_sample_size,4);            // matrix of horizontal vectors Mean
      static matrix  ms(m_sample_size,4);            // matrix of horizontal vectors Std
      static matrix  x_norm(m_sample_size,4);        // matrix for prices normalize
   
      //--- prepare input data
      matrix rates;
      //--- request last bars
      date-=date%PeriodSeconds(m_period);
      if(!rates.CopyRates(m_symbol,m_period,COPY_RATES_OHLC,date-1,m_sample_size))
         return(DBL_MAX);
      //--- get series Mean
      vector m=rates.Mean(1);
      //--- get series Std
      vector s=rates.Std(1);
      //--- prepare matrices for prices normalization
      for(int i=0; i<m_sample_size; i++)
        {
         mm.Row(m,i);
         ms.Row(s,i);
        }
      //--- the input of the model must be a set of vertical OHLC vectors
      x_norm=rates.Transpose();
      //--- normalize prices
      x_norm-=mm;
      x_norm/=ms;
   
      //--- run the inference
      input_data.Assign(x_norm);
      if(!OnnxRun(m_handle,ONNX_NO_CONVERSION,input_data,output_data))
         return(DBL_MAX);

      //--- denormalize the price from the output value
      double predicted=output_data[0]*s[3]+m[3];
      //--- return prediction
      return(predicted);
     }
  };
//+------------------------------------------------------------------+

This model is similar to our very first model published in the public project MQL5\Shared Projects\ONNX.Price.Prediction.

The series of 10 OHLC prices should be normalized in the same way as during training, namely, the deviation from the average price in the series is divided by the standard deviation in the series. Thus, we put the series into a certain range with a mean of 0 and a spread of 1, which improves convergence during training.


D1_30 model class

The second model is called model.eurusd.D1.30.onnx. The regression model trained on EURUSD D1 on the series of 30 Close prices and two simple moving averages with averaging periods of 21 and 34. 

//+------------------------------------------------------------------+
//|                                             ModelEurusdD1_30.mqh |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#include "ModelSymbolPeriod.mqh"

#resource "Python/model.eurusd.D1.30.onnx" as uchar model_eurusd_D1_30[]

//+------------------------------------------------------------------+
//| ONNX-model wrapper class                                         |
//+------------------------------------------------------------------+
class CModelEurusdD1_30 : public CModelSymbolPeriod
  {
private:
   int               m_sample_size;
   int               m_fast_period;
   int               m_slow_period;
   int               m_sma_fast;
   int               m_sma_slow;

public:
   //+------------------------------------------------------------------+
   //| Constructor                                                      |
   //+------------------------------------------------------------------+
   CModelEurusdD1_30(void) : CModelSymbolPeriod("EURUSD",PERIOD_D1)
     {
      m_name="D1_30";
      m_sample_size=30;
      m_fast_period=21;
      m_slow_period=34;
      m_sma_fast=INVALID_HANDLE;
      m_sma_slow=INVALID_HANDLE;
     }

   //+------------------------------------------------------------------+
   //| ONNX-model initialization                                        |
   //+------------------------------------------------------------------+
   virtual bool Init(const string symbol, const ENUM_TIMEFRAMES period)
     {
      //--- check symbol, period, create model
      if(!CModelSymbolPeriod::CheckInit(symbol,period,model_eurusd_D1_30))
        {
         Print("model_eurusd_D1_30 : initialization error");
         return(false);
        }

      //--- since not all sizes defined in the input tensor we must set them explicitly
      //--- first index - batch size, second index - series size, third index - number of series (Close, MA fast, MA slow)
      const long input_shape[] = {1,m_sample_size,3};
      if(!OnnxSetInputShape(m_handle,0,input_shape))
        {
         Print("model_eurusd_D1_30 : OnnxSetInputShape error ",GetLastError());
         return(false);
        }
   
      //--- since not all sizes defined in the output tensor we must set them explicitly
      //--- first index - batch size, must match the batch size of the input tensor
      //--- second index - number of predicted prices
      const long output_shape[] = {1,1};
      if(!OnnxSetOutputShape(m_handle,0,output_shape))
        {
         Print("model_eurusd_D1_30 : OnnxSetOutputShape error ",GetLastError());
         return(false);
        }
      //--- indicators
      m_sma_fast=iMA(m_symbol,m_period,m_fast_period,0,MODE_SMA,PRICE_CLOSE);
      m_sma_slow=iMA(m_symbol,m_period,m_slow_period,0,MODE_SMA,PRICE_CLOSE);
      if(m_sma_fast==INVALID_HANDLE || m_sma_slow==INVALID_HANDLE)
        {
         Print("model_eurusd_D1_30 : cannot create indicator");
         return(false);
        }
      //--- ok
      return(true);
     }

   //+------------------------------------------------------------------+
   //| Predict price                                                    |
   //+------------------------------------------------------------------+
   virtual double PredictPrice(datetime date)
     {
      static matrixf input_data(m_sample_size,3);    // matrix for prepared input data
      static vectorf output_data(1);                 // vector to get result
      static matrix  x_norm(m_sample_size,3);        // matrix for prices normalize
      static vector  vtemp(m_sample_size);
      static double  ma_buffer[];
   
      //--- request last bars
      date-=date%PeriodSeconds(m_period);
      if(!vtemp.CopyRates(m_symbol,m_period,COPY_RATES_CLOSE,date-1,m_sample_size))
         return(DBL_MAX);
      //--- get series Mean
      double m=vtemp.Mean();
      //--- get series Std
      double s=vtemp.Std();
      //--- normalize
      vtemp-=m;
      vtemp/=s;
      x_norm.Col(vtemp,0);
      //--- fast sma
      if(CopyBuffer(m_sma_fast,0,date-1,m_sample_size,ma_buffer)!=m_sample_size)
         return(-1);
      vtemp.Assign(ma_buffer);
      m=vtemp.Mean();
      s=vtemp.Std();
      vtemp-=m;
      vtemp/=s;
      x_norm.Col(vtemp,1);
      //--- slow sma
      if(CopyBuffer(m_sma_slow,0,date-1,m_sample_size,ma_buffer)!=m_sample_size)
         return(-1);
      vtemp.Assign(ma_buffer);
      m=vtemp.Mean();
      s=vtemp.Std();
      vtemp-=m;
      vtemp/=s;
      x_norm.Col(vtemp,2);
   
      //--- run the inference
      input_data.Assign(x_norm);
      if(!OnnxRun(m_handle,ONNX_NO_CONVERSION,input_data,output_data))
         return(DBL_MAX);

      //--- denormalize the price from the output value
      double predicted=output_data[0]*s+m;
      //--- return prediction
      return(predicted);
     }
  };
//+------------------------------------------------------------------+

As in the previous class, the CheckInit base class method is called in the Init method. In the base class method, a session is created for the ONNX model and the sizes of the input and output tensors are explicitly set.

PredictPrice method provides the series of 30 previous Close and calculated moving averages. The data is normalized in the same way as in training.

The model was developed for the article "Wrapping ONNX models in classes" and converted from classification to regression for this article.


D1_52 model class

The third model is called model.eurusd.D1.52.onnx. The regression model trained on EURUSD D1 on the series of 52 Close prices. 

//+------------------------------------------------------------------+
//|                                             ModelEurusdD1_52.mqh |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#include "ModelSymbolPeriod.mqh"

#resource "Python/model.eurusd.D1.52.onnx" as uchar model_eurusd_D1_52[]

//+------------------------------------------------------------------+
//| ONNX-model wrapper class                                         |
//+------------------------------------------------------------------+
class CModelEurusdD1_52 : public CModelSymbolPeriod
  {
private:
   int               m_sample_size;

public:
   //+------------------------------------------------------------------+
   //| Constructor                                                      |
   //+------------------------------------------------------------------+
   CModelEurusdD1_52(void) : CModelSymbolPeriod("EURUSD",PERIOD_D1,0.0001)
     {
      m_name="D1_52";
      m_sample_size=52;
     }

   //+------------------------------------------------------------------+
   //| ONNX-model initialization                                        |
   //+------------------------------------------------------------------+
   virtual bool Init(const string symbol, const ENUM_TIMEFRAMES period)
     {
      //--- check symbol, period, create model
      if(!CModelSymbolPeriod::CheckInit(symbol,period,model_eurusd_D1_52))
        {
         Print("model_eurusd_D1_52 : initialization error");
         return(false);
        }

      //--- since not all sizes defined in the input tensor we must set them explicitly
      //--- first index - batch size, second index - series size, third index - number of series (only Close)
      const long input_shape[] = {1,m_sample_size,1};
      if(!OnnxSetInputShape(m_handle,0,input_shape))
        {
         Print("model_eurusd_D1_52 : OnnxSetInputShape error ",GetLastError());
         return(false);
        }
   
      //--- since not all sizes defined in the output tensor we must set them explicitly
      //--- first index - batch size, must match the batch size of the input tensor
      //--- second index - number of predicted prices (we only predict Close)
      const long output_shape[] = {1,1};
      if(!OnnxSetOutputShape(m_handle,0,output_shape))
        {
         Print("model_eurusd_D1_52 : OnnxSetOutputShape error ",GetLastError());
         return(false);
        }
      //--- ok
      return(true);
     }

   //+------------------------------------------------------------------+
   //| Predict price                                                    |
   //+------------------------------------------------------------------+
   virtual double PredictPrice(datetime date)
     {
      static vectorf output_data(1);            // vector to get result
      static vector  x_norm(m_sample_size);     // vector for prices normalize
   
      //--- set date to day begin
      date-=date%PeriodSeconds(m_period);
      //--- check for calculate min and max
      double price_min=0;
      double price_max=0;
      GetMinMaxClose(date,price_min,price_max);
      //--- check for normalization possibility
      if(price_min>=price_max)
         return(DBL_MAX);
      //--- request last bars
      if(!x_norm.CopyRates(m_symbol,m_period,COPY_RATES_CLOSE,date-1,m_sample_size))
         return(DBL_MAX);
      //--- normalize prices
      x_norm-=price_min;
      x_norm/=(price_max-price_min);
      //--- run the inference
      if(!OnnxRun(m_handle,ONNX_DEFAULT,x_norm,output_data))
         return(DBL_MAX);

      //--- denormalize the price from the output value
      double predicted=output_data[0]*(price_max-price_min)+price_min;
      //--- return prediction
      return(predicted);
     }

private:
   //+------------------------------------------------------------------+
   //| Get minimal and maximal Close for last 52 weeks                  |
   //+------------------------------------------------------------------+
   void GetMinMaxClose(const datetime date,double& price_min,double& price_max)
     {
      static vector close;

      close.CopyRates(m_symbol,m_period,COPY_RATES_CLOSE,date,m_sample_size*7+1);
      price_min=close.Min();
      price_max=close.Max();
     }
  };
//+------------------------------------------------------------------+

The price normalization before submitting the model differs from the previous ones. MinMaxScaler was used during training. Therefore, we take the minimum and maximum prices for the period of 52 weeks before the forecast date.

The model is similar to the one described in the article "How to use ONNX models in MQL5".


D1_63 model class

Finally, the fourth model is called model.eurusd.D1.63.onnx. The regression model trained on EURUSD D1 on the series of 63 Close prices.

//+------------------------------------------------------------------+
//|                                             ModelEurusdD1_63.mqh |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#include "ModelSymbolPeriod.mqh"

#resource "Python/model.eurusd.D1.63.onnx" as uchar model_eurusd_D1_63[]

//+------------------------------------------------------------------+
//| ONNX-model wrapper class                                         |
//+------------------------------------------------------------------+
class CModelEurusdD1_63 : public CModelSymbolPeriod
  {
private:
   int               m_sample_size;

public:
   //+------------------------------------------------------------------+
   //| Constructor                                                      |
   //+------------------------------------------------------------------+
   CModelEurusdD1_63(void) : CModelSymbolPeriod("EURUSD",PERIOD_D1)
     {
      m_name="D1_63";
      m_sample_size=63;
     }

   //+------------------------------------------------------------------+
   //| ONNX-model initialization                                        |
   //+------------------------------------------------------------------+
   virtual bool Init(const string symbol, const ENUM_TIMEFRAMES period)
     {
      //--- check symbol, period, create model
      if(!CModelSymbolPeriod::CheckInit(symbol,period,model_eurusd_D1_63))
        {
         Print("model_eurusd_D1_63 : initialization error");
         return(false);
        }

      //--- since not all sizes defined in the input tensor we must set them explicitly
      //--- first index - batch size, second index - series size
      const long input_shape[] = {1,m_sample_size};
      if(!OnnxSetInputShape(m_handle,0,input_shape))
        {
         Print("model_eurusd_D1_63 : OnnxSetInputShape error ",GetLastError());
         return(false);
        }
   
      //--- since not all sizes defined in the output tensor we must set them explicitly
      //--- first index - batch size, must match the batch size of the input tensor
      //--- second index - number of predicted prices
      const long output_shape[] = {1,1};
      if(!OnnxSetOutputShape(m_handle,0,output_shape))
        {
         Print("model_eurusd_D1_63 : OnnxSetOutputShape error ",GetLastError());
         return(false);
        }
      //--- ok
      return(true);
     }

   //+------------------------------------------------------------------+
   //| Predict price                                                    |
   //+------------------------------------------------------------------+
   virtual double PredictPrice(datetime date)
     {
      static vectorf input_data(m_sample_size);  // vector for prepared input data
      static vectorf output_data(1);             // vector to get result
   
      //--- request last bars
      date-=date%PeriodSeconds(m_period);
      if(!input_data.CopyRates(m_symbol,m_period,COPY_RATES_CLOSE,date-1,m_sample_size))
         return(DBL_MAX);
      //--- get series Mean
      float m=input_data.Mean();
      //--- get series Std
      float s=input_data.Std();
      //--- normalize prices
      input_data-=m;
      input_data/=s;
   
      //--- run the inference
      if(!OnnxRun(m_handle,ONNX_NO_CONVERSION,input_data,output_data))
         return(DBL_MAX);

      //--- denormalize the price from the output value
      double predicted=output_data[0]*s+m;
      //--- return prediction
      return(predicted);
     }
  };
//+------------------------------------------------------------------+

PredictPrice method provides the series of 63 previous Close. The data is normalized in the same way as in the first and second models.

The model has already been developed for the article "An example of how to ensemble ONNX models in MQL5".


Combining all models in one script. Reality, predictions and regression metrics

In order to apply regression metrics, we should make a certain number of predictions (vector_pred) and take actual data for the same dates (vector_true).

Since all our models are wrapped in classes that derive from the same base class, we can evaluate them all at once.

The script is very simple

//+------------------------------------------------------------------+
//|                                    ONNX.eurusd.D1.4M.Metrics.mq5 |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2023, MetaQuotes Ltd."
#property link      "https://www.mql5.com"
#property version   "1.00"

#define MODELS 4
#include "ModelEurusdD1_10.mqh"
#include "ModelEurusdD1_30.mqh"
#include "ModelEurusdD1_52.mqh"
#include "ModelEurusdD1_63.mqh"

#property script_show_inputs
input datetime InpStartDate = D'2023.01.01';
input datetime InpStopDate  = D'2023.01.31';

CModelSymbolPeriod *ExtModels[MODELS];

struct PredictedPrices
  {
   string            model;
   double            pred[];
  };
PredictedPrices ExtPredicted[MODELS];

double ExtClose[];

struct Metrics
  {
   string            model;
   double            mae;
   double            mse;
   double            rmse;
   double            r2;
   double            mape;
   double            mspe;
   double            rmsle;
  };
Metrics ExtMetrics[MODELS];

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//--- init section
   if(!Init())
      return;

//--- predictions test loop
   datetime dates[];
   if(CopyTime(_Symbol,_Period,InpStartDate,InpStopDate,dates)<=0)
     {
      Print("Cannot get data from ",InpStartDate," to ",InpStopDate);
      return;
     }
   for(uint n=0; n<dates.Size(); n++)
      GetPredictions(dates[n]);
      
   CopyClose(_Symbol,_Period,InpStartDate,InpStopDate,ExtClose);
   CalculateMetrics();

//--- deinit section
   Deinit();
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
bool Init()
  {
   ExtModels[0]=new CModelEurusdD1_10;
   ExtModels[1]=new CModelEurusdD1_30;
   ExtModels[2]=new CModelEurusdD1_52;
   ExtModels[3]=new CModelEurusdD1_63;

   for(long i=0; i<ExtModels.Size(); i++)
     {
      if(!ExtModels[i].Init(_Symbol,_Period))
        {
         Deinit();
         return(false);
        }
     }

   for(long i=0; i<ExtModels.Size(); i++)
      ExtPredicted[i].model=ExtModels[i].GetModelName();

   return(true);
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void Deinit()
  {
   for(uint i=0; i<ExtModels.Size(); i++)
      delete ExtModels[i];
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void GetPredictions(datetime date)
  {
//--- collect predicted prices
   for(uint i=0; i<ExtModels.Size(); i++)
      ExtPredicted[i].pred.Push(ExtModels[i].PredictPrice(date));
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void CalculateMetrics()
  {
   vector vector_pred,vector_true;
   vector_true.Assign(ExtClose);

   for(uint i=0; i<ExtModels.Size(); i++)
     {
      ExtMetrics[i].model=ExtPredicted[i].model;
      vector_pred.Assign(ExtPredicted[i].pred);
      ExtMetrics[i].mae  =vector_pred.RegressionMetric(vector_true,REGRESSION_MAE);
      ExtMetrics[i].mse  =vector_pred.RegressionMetric(vector_true,REGRESSION_MSE);
      ExtMetrics[i].rmse =vector_pred.RegressionMetric(vector_true,REGRESSION_RMSE);
      ExtMetrics[i].r2   =vector_pred.RegressionMetric(vector_true,REGRESSION_R2);
      ExtMetrics[i].mape =vector_pred.RegressionMetric(vector_true,REGRESSION_MAPE);
      ExtMetrics[i].mspe =vector_pred.RegressionMetric(vector_true,REGRESSION_MSPE);
      ExtMetrics[i].rmsle=vector_pred.RegressionMetric(vector_true,REGRESSION_RMSLE);
     }

   ArrayPrint(ExtMetrics);
  }
//+------------------------------------------------------------------+

Let's run the script on EURUSD D1 chart and set the dates from January 1 to January 31, 2023 inclusive. What do we see?

    [model]   [mae]   [mse]  [rmse]     [r2]  [mape]  [mspe] [rmsle]
[0] "D1_10" 0.00381 0.00003 0.00530  0.77720 0.00356 0.00002 0.00257
[1] "D1_30" 0.01809 0.00039 0.01963 -2.05545 0.01680 0.00033 0.00952
[2] "D1_52" 0.00472 0.00004 0.00642  0.67327 0.00440 0.00004 0.00311
[3] "D1_63" 0.00413 0.00003 0.00559  0.75230 0.00385 0.00003 0.00270

The negative R-squared value is immediately noticeable in the second row. This means the model is not working. It is interesting to look at the prediction graphs.

Four model forecasts

We see the chart D1_30 far away from actual Close prices and other forecasts. None of the metrics for this model are encouraging. MAE shows the forecast accuracy of 1809 price points! Keep in mind, however, that the model was initially developed for the previous article as a classification model, not a regression one. The example is pretty clear.

Let's consider other models separately.

The first candidate for analysis is D1_10

    [model]   [mae]   [mse]  [rmse]     [r2]  [mape]  [mspe] [rmsle]
[0] "D1_10" 0.00381 0.00003 0.00530  0.77720 0.00356 0.00002 0.00257

Let's look at the chart of prices predicted by this model.

D1_10 forecast

The RMSLE metric does not make much sense, since the spread from 1.05 to 1.09 is much less than one order of magnitude. The MAPE and MSPE metrics are close in their values to MAE and MSE due to the peculiarities of the EURUSD exchange rate as it is close to one. However, when calculating percentage deviations there is a nuance that is not present when calculating absolute deviations.

MAPE = |(y_true-y_pred)/y_true|

if y_true = 10 and y_pred = 5
MAPE = 0.5

if y_true = 5 and y_pred = 10
MAPE = 1.0

In other words, this metric (like MSPE) is asymmetrical. This means that in the case where the forecast is higher than the fact, we get a larger error.

Good result of the R-squared metric has been achieved for the simple model cobbled together for purely methodological purposes, namely to show how you can work with ONNX models in MQL5.


Second candidate - D1_63

    [model]   [mae]   [mse]  [rmse]     [r2]  [mape]  [mspe] [rmsle]
[3] "D1_63" 0.00413 0.00003 0.00559  0.75230 0.00385 0.00003 0.00270

D1_63 forecast

The forecast is very similar to the previous one visually. Metric values confirm similarity

[0] "D1_10" 0.00381 0.00003 0.00530  0.77720 0.00356 0.00002 0.00257
[3] "D1_63" 0.00413 0.00003 0.00559  0.75230 0.00385 0.00003 0.00270

Next we will see which of these models will perform better in the tester during the same period.


Now for D1_52

    [model]   [mae]   [mse]  [rmse]     [r2]  [mape]  [mspe] [rmsle]
[2] "D1_52" 0.00472 0.00004 0.00642  0.67327 0.00440 0.00004 0.00311

We consider it only because its R-square is greater than 0.5

D1_52 forecast

Almost all predicted prices are below the Close chart, as in our worst case. Despite comparable metric values to those of the previous two models, this model does not inspire any optimism. We will check this in the next paragraph.


Running ONNX models in the tester

Below is a very simple EA for checking our models in the tester

//+------------------------------------------------------------------+
//|                                    ONNX.eurusd.D1.Prediction.mq5 |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright   "Copyright 2023, MetaQuotes Ltd."
#property link        "https://www.mql5.com"
#property version     "1.00"

#include "ModelEurusdD1_10.mqh"
#include "ModelEurusdD1_30.mqh"
#include "ModelEurusdD1_52.mqh"
#include "ModelEurusdD1_63.mqh"
#include <Trade\Trade.mqh>

input double InpLots = 1.0;    // Lots amount to open position

//CModelEurusdD1_10 ExtModel;
//CModelEurusdD1_30 ExtModel;
CModelEurusdD1_52 ExtModel;
//CModelEurusdD1_63 ExtModel;
CTrade            ExtTrade;

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   if(!ExtModel.Init(_Symbol,_Period))
      return(INIT_FAILED);
   Print("model ",ExtModel.GetModelName());
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
   ExtModel.Shutdown();
  }
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
   if(!ExtModel.CheckOnTick())
      return;

//--- predict next price movement
   vector prob(3);
   int    predicted_class=ExtModel.PredictClass(TimeCurrent(),prob);
   Print("predicted class ",predicted_class);
//--- check trading according to prediction
   if(predicted_class>=0)
      if(PositionSelect(_Symbol))
         CheckForClose(predicted_class);
      else
         CheckForOpen(predicted_class);
  }
//+------------------------------------------------------------------+
//| Check for open position conditions                               |
//+------------------------------------------------------------------+
void CheckForOpen(const int predicted_class)
  {
   ENUM_ORDER_TYPE signal=WRONG_VALUE;
//--- check signals
   if(predicted_class==PRICE_DOWN)
      signal=ORDER_TYPE_SELL;    // sell condition
   else
     {
      if(predicted_class==PRICE_UP)
         signal=ORDER_TYPE_BUY;  // buy condition
     }

//--- open position if possible according to signal
   if(signal!=WRONG_VALUE && TerminalInfoInteger(TERMINAL_TRADE_ALLOWED))
     {
      double price=SymbolInfoDouble(_Symbol,(signal==ORDER_TYPE_SELL) ? SYMBOL_BID : SYMBOL_ASK);
      ExtTrade.PositionOpen(_Symbol,signal,InpLots,price,0,0);
     }
  }
//+------------------------------------------------------------------+
//| Check for close position conditions                              |
//+------------------------------------------------------------------+
void CheckForClose(const int predicted_class)
  {
   bool bsignal=false;
//--- position already selected before
   long type=PositionGetInteger(POSITION_TYPE);
//--- check signals
   if(type==POSITION_TYPE_BUY && predicted_class==PRICE_DOWN)
      bsignal=true;
   if(type==POSITION_TYPE_SELL && predicted_class==PRICE_UP)
      bsignal=true;

//--- close position if possible
   if(bsignal && TerminalInfoInteger(TERMINAL_TRADE_ALLOWED))
     {
      ExtTrade.PositionClose(_Symbol,3);
      //--- open opposite
      CheckForOpen(predicted_class);
     }
  }
//+------------------------------------------------------------------+

Indeed, according to the D1_52 model, only one sell trade was opened and the trend, according to this model, did not change throughout the entire testing period

Testing D1_52


2023.06.09 16:18:31.967 Symbols EURUSD: symbol to be synchronized
2023.06.09 16:18:31.968 Symbols EURUSD: symbol synchronized, 3720 bytes of symbol info received
2023.06.09 16:18:32.023 History EURUSD: load 27 bytes of history data to synchronize in 0:00:00.001
2023.06.09 16:18:32.023 History EURUSD: history synchronized from 2011.01.03 to 2023.04.07
2023.06.09 16:18:32.124 History EURUSD,Daily: history cache allocated for 283 bars and contains 260 bars from 2022.01.03 00:00 to 2022.12.30 00:00
2023.06.09 16:18:32.124 History EURUSD,Daily: history begins from 2022.01.03 00:00
2023.06.09 16:18:32.126 Tester  EURUSD,Daily (MetaQuotes-Demo): 1 minutes OHLC ticks generating
2023.06.09 16:18:32.126 Tester  EURUSD,Daily: testing of Experts\article_2\ONNX.eurusd.D1.Prediction.ex5 from 2023.01.01 00:00 to 2023.02.01 00:00 started with inputs:
2023.06.09 16:18:32.126 Tester    InpLots=1.0
2023.06.09 16:18:32.161 ONNX    api version 1.16.0 initialized
2023.06.09 16:18:32.180 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.01 00:00:00   model D1_52
2023.06.09 16:18:32.194 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.02 07:02:00   predicted class 2
2023.06.09 16:18:32.194 Trade   2023.01.02 07:02:00   instant sell 1 EURUSD at 1.07016 (1.07016 / 1.07023 / 1.07016)
2023.06.09 16:18:32.194 Trades  2023.01.02 07:02:00   deal #2 sell 1 EURUSD at 1.07016 done (based on order #2)
2023.06.09 16:18:32.194 Trade   2023.01.02 07:02:00   deal performed [#2 sell 1 EURUSD at 1.07016]
2023.06.09 16:18:32.194 Trade   2023.01.02 07:02:00   order performed sell 1 at 1.07016 [#2 sell 1 EURUSD at 1.07016]
2023.06.09 16:18:32.195 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.02 07:02:00   CTrade::OrderSend: instant sell 1.00 EURUSD at 1.07016 [done at 1.07016]
2023.06.09 16:18:32.196 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.03 00:00:00   predicted class 2
2023.06.09 16:18:32.199 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.04 00:00:00   predicted class 2
2023.06.09 16:18:32.201 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.05 00:00:30   predicted class 2
2023.06.09 16:18:32.203 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.06 00:00:00   predicted class 2
2023.06.09 16:18:32.206 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.09 00:02:00   predicted class 2
2023.06.09 16:18:32.208 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.10 00:00:00   predicted class 2
2023.06.09 16:18:32.210 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.11 00:00:00   predicted class 2
2023.06.09 16:18:32.213 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.12 00:00:00   predicted class 2
2023.06.09 16:18:32.215 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.13 00:00:00   predicted class 2
2023.06.09 16:18:32.217 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.16 00:03:00   predicted class 2
2023.06.09 16:18:32.220 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.17 00:00:00   predicted class 2
2023.06.09 16:18:32.222 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.18 00:00:30   predicted class 2
2023.06.09 16:18:32.224 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.19 00:00:00   predicted class 2
2023.06.09 16:18:32.227 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.20 00:00:30   predicted class 2
2023.06.09 16:18:32.229 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.23 00:02:00   predicted class 2
2023.06.09 16:18:32.231 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.24 00:00:00   predicted class 2
2023.06.09 16:18:32.234 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.25 00:00:00   predicted class 2
2023.06.09 16:18:32.236 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.26 00:00:00   predicted class 2
2023.06.09 16:18:32.238 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.27 00:00:00   predicted class 2
2023.06.09 16:18:32.241 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.30 00:03:00   predicted class 2
2023.06.09 16:18:32.243 ONNX.eurusd.D1.Prediction (EURUSD,D1)   2023.01.31 00:00:00   predicted class 2
2023.06.09 16:18:32.245 Trade   2023.01.31 23:59:59   position closed due end of test at 1.08621 [#2 sell 1 EURUSD 1.07016]
2023.06.09 16:18:32.245 Trades  2023.01.31 23:59:59   deal #3 buy 1 EURUSD at 1.08621 done (based on order #3)
2023.06.09 16:18:32.245 Trade   2023.01.31 23:59:59   deal performed [#3 buy 1 EURUSD at 1.08621]
2023.06.09 16:18:32.245 Trade   2023.01.31 23:59:59   order performed buy 1 at 1.08621 [#3 buy 1 EURUSD at 1.08621]
2023.06.09 16:18:32.245 Tester  final balance 8366.00 USD
2023.06.09 16:18:32.249 Tester  EURUSD,Daily: 123499 ticks, 22 bars generated. Environment synchronized in 0:00:00.043. Test passed in 0:00:00.294 (including ticks preprocessing 0:00:00.016).

As mentioned in the previous section, the D1_52 model does not inspire optimism. This is confirmed by the test results.

Let's change just two lines of code

#include "ModelEurusdD1_10.mqh"
#include "ModelEurusdD1_30.mqh"
#include "ModelEurusdD1_52.mqh"
#include "ModelEurusdD1_63.mqh"
#include <Trade\Trade.mqh>

input double InpLots = 1.0;    // Lots amount to open position

CModelEurusdD1_10 ExtModel;
//CModelEurusdD1_30 ExtModel;
//CModelEurusdD1_52 ExtModel;
//CModelEurusdD1_63 ExtModel;
CTrade            ExtTrade;

and launch the model D1_10 for testing.

D1_10 test results

The results are good. The test graph is also promising.

D1_10 test graph


Let's fix two lines of code again and test the D1_63 model.

D1_63 test results

The graph.

D1_63 test graph

The test graph is much worse than that of the D1_10 model.

Comparing two models D1_10 and D1_63, we can see that the first model has better regression metrics than the second one. The tester shows the same thing.

Important note: Please be advised that the models used in the article are presented only to demonstrate how to work with ONNX models using the MQL5 language. The Expert Advisor is not intended for trading on real accounts.


Conclusion

The most appropriate metric for evaluating price prediction models is R-squared. Considering the aggregate of MAE - RMSE - MAPE can be very useful. RMSLE metric may not be considered in price forecasting tasks. It is very useful to have several models for evaluation, even if it is the same model with modifications.

We understand that a sample of 22 values is insufficient for serious research, but it was not our intention to do a statistical study. We have provided only the use case instead.


Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/12772

Attached files |
Download ZIP
MQL5.zip (1005.2 KB)
Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.

Other articles by this author

Go to discussion
Estimate future performance with confidence intervals Estimate future performance with confidence intervals
In this article we delve into the application of boostrapping techniques as a means to estimate the future performance of an automated strategy.
Category Theory in MQL5 (Part 20): A detour to Self-Attention and the Transformer Category Theory in MQL5 (Part 20): A detour to Self-Attention and the Transformer
We digress in our series by pondering at part of the algorithm to chatGPT. Are there any similarities or concepts borrowed from natural transformations? We attempt to answer these and other questions in a fun piece, with our code in a signal class format.
Category Theory in MQL5 (Part 21): Natural Transformations with LDA Category Theory in MQL5 (Part 21): Natural Transformations with LDA
This article, the 21st in our series, continues with a look at Natural Transformations and how they can be implemented using linear discriminant analysis. We present applications of this in a signal class format, like in the previous article.
Developing an MQTT client for MetaTrader 5: a TDD approach — Part 2 Developing an MQTT client for MetaTrader 5: a TDD approach — Part 2
This article is part of a series describing our development steps of a native MQL5 client for the MQTT protocol. In this part we describe our code organization, the first header files and classes, and how we are writing our tests. This article also includes brief notes about the Test-Driven-Development practice and how we are applying it to this project.