New MetaTrader 5 build 3620: Web Terminal improvements, ONNX support and fast matrix multiplications in MQL5

MetaQuotes 2023.03.03 10:14

The new version of the updated MetaTrader 5 platform will be released on Friday, March 10, 2023.

The update provides web terminal improvements. We have implemented a set of color templates for the web terminal interface and have enhanced the symbol specification window.

New MetaTrader 5 build 3620: Web Terminal improvements and fast matrix multiplications in MQL5

Also, the new version provides General Matrix Multiplication (GeMM) in MQL5. This algorithm speeds up calculations on most processors. The new algorithm is currently supported in the matrix::GeMM method.

We have also implemented support for operations with ONNX models in MQL5. This will greatly facilitate the use of neural networks in Expert Advisors.

The new version features the following changes:

MetaTrader 5 Client Terminal build 3620

Terminal: Fixed total profit calculations in trading reports.
Terminal: Updated fundamental data for trading instruments available through the Market Watch window.
Terminal: Fixed trading platform launch under Wine 7.0.1 in Linux-systems.
Terminal: Fixed adding of symbols to the Market Depth via the search bar. A symbol found by description could not be added to the list by clicking on its line.
MQL5: Added support for operations with ONNX models (Open Neural Network Exchange).

ONNX is an open-source format for machine learning models. This format is supported by many platforms, including Chainer, Caffee2 and PyTorch. Create an ONNX model using specialized tools, integrate it into your MQL5 application and use it to make trading decisions.

Descriptions of all supported functions are available in the documentation. An example of a test ONNX model is available in public projects in MetaEditor. Find the ONNX.Price.Prediction project in "Toolbox \ Public projects" and select Join in the context menu. The project will download to your computer and will appear in the Navigator:

Compile the project and run it on EURUSD H1 to see the result.

In addition to the model and the MQL5 code which runs it, the project also includes the PricePredictionTraining.py Python script. It shows how you can create an ONNX model yourself. To run the script, install Python on your computer and the required modules from the prompt line:

python.exe -m pip install --upgrade pip
python -m pip install --upgrade tensorflow
python -m pip install --upgrade pandas
python -m pip install --upgrade scikit-learn
python -m pip install --upgrade matplotlib
python -m pip install --upgrade tqdm
python -m pip install --upgrade metatrader5
python -m pip install --upgrade onnx==1.12
python -m pip install --upgrade tf2onnx

Instructions on how to use ONNX are available in the documentation.
MQL5: Added support for General Matrix Multiplication (GeMM). This algorithm speeds up calculations on some processor types through parallelized tasks and optimized utilization of L1/L2/L3 caches. The calculation speed is comparable with popular packages such as Math Kernel Library (MKL) and OpenBLAS. Detailed comparative tests will be published soon.

The new algorithm is currently supported in the matrix::GeMM method. If your processor supports AVX and FMA instructions (most processors released after 2013 support these instructions), the algorithm will be enabled automatically.
MQL5: Added ability to transfer matrices and vectors to DLL. This enables the import of functions which utilize the relevant types, from external variables.

Matrices and vectors are passed to a DLL as a pointer to a buffer. For example, to pass a matrix of type float, the corresponding parameter of the function exported from the DLL must take a float-type buffer pointer. For example:

MQL5
```
#import "mmlib.dll"
bool sgemm(uint flags,matrix<float> &C,const matrix<float> &A,const matrix<float> &B,ulong M,ulong N,ulong K,float alpha,float beta);
#import
```
C++
```
extern "C" __declspec(dllexport) bool sgemm(UINT flags,float *C,const float *A,const float *B,UINT64 M,UINT64 N,UINT64 K,float alpha,float beta)
```
In addition to buffers, you should pass matrix and vector sizes for correct processing.

MQL5: Added new CopySeries function for copying synchronized timeseries from MqlRates into separate arrays.

The CopySeries function allows obtaining only the necessary timeseries into different specified arrays during one call, while all of timeseries data will be synchronized. This means that all values in the resulting arrays at a certain index N will belong to the same bar on the specified Symbol/Timeframe pair. Therefore, there is no need for the programmer to additionally synchronize the received timeseries by the bar opening time.

Unlike CopyRates, which returns the full set of timeseries as an MqlRates array, the CopySeries function allows obtaining specific required timeseries into separate arrays. This can be done by specifying a combination of flags to select the type of timeseries. The order of the arrays passed to the function must match the order of the fields in the MqlRates structure:

struct MqlRates
  {
   datetime time;         // period beginning time
   double   open;         // open price
   double   high;         // high price for the period
   double   low;          // low price for the period
   double   close;        // close price
   long     tick_volume;  // tick volume
   int      spread;       // spread
   long     real_volume;  // exchange volume
  }

Thus, if you need to get the values of the 'time', 'close' and 'real_volume' timeseries for the last 100 bars of the current Symbol/Timeframe, you should use the following call:

datetime  time[];
double    close[];
long      volume[];
CopySeries(NULL,0,0,100,COPY_RATES_TIME|COPY_RATES_CLOSE|COPY_RATES_VOLUME_REAL,time,close,volume);

The order of the arrays "time, close, volume" must match the order of the fields in the MqlRates structure. The order of values in the rates_mask is ignored. The mask could be as follows:

COPY_RATES_VOLUME_REAL|COPY_RATES_TIME|COPY_RATES_CLOSE

Example

//--- input parameters
input datetime InpDateFrom=D'2022.01.01 00:00:00';
input datetime InpDateTo  =D'2023.01.01 00:00:00';
input uint     InpCount   =20;
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart(void)
  {
//--- arrays to get timeseries from the Rates structure
   double   open[];
   double   close[];
   float    closef[];
   datetime time1[], time2[];
//---request close prices to a double array
   ResetLastError();
   int res1=CopySeries(NULL, PERIOD_CURRENT, 0, InpCount,
                       COPY_RATES_TIME|COPY_RATES_CLOSE, time1, close);
   PrintFormat("1. CopySeries  returns %d values. Error code=%d", res1, GetLastError());
   ArrayPrint(close);
   

//--- now also request open prices; use float array for close prices
   ResetLastError();
   int res2=CopySeries(NULL, PERIOD_CURRENT, 0, InpCount,
                       COPY_RATES_TIME|COPY_RATES_CLOSE|COPY_RATES_OPEN, time2, open, closef);
   PrintFormat("2. CopySeries  returns %d values. Error code=%d", res2, GetLastError());
   ArrayPrint(closef);
//--- compare the received data
   if((res1==res2) && (time1[0]==time2[0]))
     {
      Print("  | Time             |    Open      | Close double | Close float |");
      for(int i=0; i<10; i++)
        {
         PrintFormat("%d | %s |   %.5f    |   %.5f    |   %.5f   |",
                     i, TimeToString(time1[i]), open[i], close[i], closef[i]);
        }
     }
/*  Result
        1. CopySeries  returns 0 values. Error code=0
        [ 0] 1.06722 1.06733 1.06653 1.06520 1.06573 1.06649 1.06694 1.06675 1.06684 1.06604
        [10] 1.06514 1.06557 1.06456 1.06481 1.06414 1.06394 1.06364 1.06386 1.06239 1.06247
        2. CopySeries  returns 0 values. Error code=0
        [ 0] 1.06722 1.06733 1.06653 1.06520 1.06573 1.06649 1.06694 1.06675 1.06684 1.06604
        [10] 1.06514 1.06557 1.06456 1.06481 1.06414 1.06394 1.06364 1.06386 1.06239 1.06247
          | Time             |    Open      | Close double | Close float |
        0 | 2023.03.01 17:00 |   1.06660    |   1.06722    |   1.06722   |
        1 | 2023.03.01 18:00 |   1.06722    |   1.06733    |   1.06733   |
        2 | 2023.03.01 19:00 |   1.06734    |   1.06653    |   1.06653   |
        3 | 2023.03.01 20:00 |   1.06654    |   1.06520    |   1.06520   |
        4 | 2023.03.01 21:00 |   1.06520    |   1.06573    |   1.06573   |
        5 | 2023.03.01 22:00 |   1.06572    |   1.06649    |   1.06649   |
        6 | 2023.03.01 23:00 |   1.06649    |   1.06694    |   1.06694   |
        7 | 2023.03.02 00:00 |   1.06683    |   1.06675    |   1.06675   |
        8 | 2023.03.02 01:00 |   1.06675    |   1.06684    |   1.06684   |
        9 | 2023.03.02 02:00 |   1.06687    |   1.06604    |   1.06604   |
*/
  }

MQL5: Fixed OrderSend function operation. The function request could return an incorrect order ticket if the same account was used simultaneously on several platforms.
MQL5: Fixed import of EX5 libraries. An error occurred if the name of the imported library matched the name of the file into which it was imported.
MetaEditor: Added sending of push notifications to shared project members. The new option can notifies users of changes in project settings and files. To enable notifications, enter your MetaQuotes ID under the "Settings \ Security" section of your MQL5.community profile.
MetaEditor: Updated file icons in the Navigator. New, simpler metaphors will make them easier to understand.
Tester: Fixed an error which caused the input string parameter to be truncated if it contained the "|" character.
Fixed errors reported in crash logs.

MetaTrader 5 Web Terminal build 3620

Added ready-made color templates for the web terminal interface. The templates affect the display of chart bars and lines, and the prices in the Market Watch and account financial statements. Our design team has prepared color template presets, based on your suggestions and on traditional color combinations.
Redesigned symbol specification window. Trading instrument data has been rearranged into logical blocks for ease of viewing.
Fixed opening of real accounts via the web terminal. The server could return an error after filling out a registration form.
Fixed an error in the trading dialog. If the user closed a position by pressing the X button in the Toolbox window while the position modification dialog was open, the dialog contents were not reset. After the update, the dialog will automatically reset to a new order placing mode in this case.
Fixed display of the Server field in the account management dialog.
Fixed display of the current timeframe on the toolbar.
Fixed display of volumes in terms of underlying asset units, in the trading dialog.
Fixed modification of Stop Loss and Take Profit levels. Modifying one of the values could reset the second one under certain conditions.
Fixed display of investment risk warnings.

The update will be available through the Live Update system.

Mihai Baciu 2023.03.12 11:50 #1

Hello everyone! Thank you for ONNX!

I tried to open my ONNX model in MQL5 code. I used the example from the documentation: https://www.mql5.com/en/docs/onnx/onnxrun.

I got an error in the OnnxSetInputShape method. The error code was 4024.

I could not find the error description here: https://www.mql5.com/en/docs/constants/errorswarnings/errorcodes

I made sure, that my input/output shapes are completely fixed (without -1 dimensions). Input shape: (1, 90, 4), output shape: (1, 1).

What does the error code mean?

ONNX input and output Need Help Regarding ONNX Mql5 error when use

Carl Schreiber 2023.03.12 12:00 #2

Well it's new and maybe not yet completed - go through the function with the debugger for more information.

hffans27 2023.03.12 12:17 #3

Mihai Baciu #:

Hello everyone! Thank you for ONNX!

I tried to open my ONNX model in MQL5 code. I used the example from the documentation: https://www.mql5.com/en/docs/onnx/onnxrun.

I got an error in the OnnxSetInputShape method. The error code was 4024.

I could not find the error description here: https://www.mql5.com/en/docs/constants/errorswarnings/errorcodes

I made sure, that my input/output shapes are completely fixed (without -1 dimensions). Input shape: (1, 90, 4), output shape: (1, 1).

What does the error code mean?

Hi， i am a newbie want to load my onnx model to mql5 and use it to predict. i never use onnx before and get confused about its shape expression.

in your shapes, your input shape is (1,90,4), could you kindly explain what does these 3 digits meaning? i guess 90 is features count, and what else? thank you!

ONNX input and output Need Help Regarding ONNX OnnxRun failure

MetaQuotes 2023.03.12 23:21 #4

Mihai Baciu #:

Hello everyone! Thank you for ONNX!

I tried to open my ONNX model in MQL5 code. I used the example from the documentation: https://www.mql5.com/en/docs/onnx/onnxrun.

I got an error in the OnnxSetInputShape method. The error code was 4024.

I could not find the error description here: https://www.mql5.com/en/docs/constants/errorswarnings/errorcodes

I made sure, that my input/output shapes are completely fixed (without -1 dimensions). Input shape: (1, 90, 4), output shape: (1, 1).

What does the error code mean?

Join ONNX.Price.Prediction projects and test it, please:

Mihai Baciu 2023.03.13 18:39 #5

MetaQuotes #:

Join ONNX.Price.Prediction projects and test it, please:

Could you please just tell what the error code 4024 mean? The developers must know, ask them. They wrote the code, but didn't document it here: https://www.mql5.com/en/docs/constants/errorswarnings/errorcodes

Mihai Baciu 2023.03.13 18:44 #6

hffans27 #:

Hi， i am a newbie want to load my onnx model to mql5 and use it to predict. i never use onnx before and get confused about its shape expression.

in your shapes, your input shape is (1,90,4), could you kindly explain what does these 3 digits meaning? i guess 90 is features count, and what else? thank you!

The first dimension is called "batch size". It is the number of objects for which the model makes predictions. In this example I provide just 1 object.

The second dimension is called "sequence length". How many bars are used to make a prediction. In this case we take 90 bars.

The third dimension equals 4 because each bar consists of 4 numbers: Open, High, Low and Close prices.

So, it can be read like this: (1 object, 90 bars, 4 prices).

A newcomer asks! Discussion of article "Testing Candlestick analysis. Pattern coding

Mihai Baciu 2023.03.14 20:04 #7

Hello, MetaQuotes moderators! Could you please contact your developers and ask them what the error code 4024 mean? Just ask them and post the answer here, please.

Sergey Golubev 2023.03.14 20:12 #8

Mihai Baciu #:
Hello, MetaQuotes moderators! Could you please contact your developers and ask them what the error code 4024 mean? Just ask them and post the answer here, please.

use search (the search results are here).

Alain Verleyen 2023.03.14 21:12 #9

Mihai Baciu #:
Hello, MetaQuotes moderators! Could you please contact your developers and ask them what the error code 4024 mean? Just ask them and post the answer here, please.

It's undocumented "Internal error". I will try to get more details according to your context.

hffans27 2023.03.15 06:15 #10

Mihai Baciu #:

The first dimension is called "batch size". It is the number of objects for which the model makes predictions. In this example I provide just 1 object.

The second dimension is called "sequence length". How many bars are used to make a prediction. In this case we take 90 bars.

The third dimension equals 4 because each bar consists of 4 numbers: Open, High, Low and Close prices.

So, it can be read like this: (1 object, 90 bars, 4 prices).

thank you and it's totally different(2&3) from what I thought.

Could you help me to see this situation? in Mql5 , i use bar data(OHLC) and some indicators to generate 25 features in every 1-minute bar generate event by running an EA. the EA ran from 2023.01.04 01:00-11:00 (10 min) in history debug mode, so EA eventually output 10*60=600 samples(rows) and 25 features (columns) table, after training a model for this 600*25 table in Pyhton, i got a model and then convert to onnx format.

so for this model, how should set input shape and output shape in mql5? for 1 dimension i should take 1, because in realtime prediction ,every 1 minute i need model to predict ONE record.

actually what i coufused about onnx is why it has to specify the input and output data shapes for a model.

in example i posted above, for a sample, the model always receives 25 features and outputs a scalar value of prediction probability. So input shape is always (1,25,1 or 1,1,25 or 1,25) and output shape always (1,1). Since features count is unchanged, why is there a OnnxSetInputShape method? Can it work normally if I set it to 24? and in the same way , the output shape is always (1,1) ，this is not what I set, it is determined by the shape of the training set when the model is trained.

Documentation on MQL5: ONNX models / OnnxSetInputShape

www.mql5.com

OnnxSetInputShape - ONNX models - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

Machine learning in trading: Discussion of article "How Machine learning for robots

1 2 3

New comment