Machine learning in trading: theory, models, practice and algo-trading - page 2938

 
Aleksey Nikolayev #:

Right click on the project name. It should be like this:


Yes, there is such a thing, but it writes an error
2023.02.28 13:05:45.151 Storage checkout started for MQL5\Shared Projects\ONNX.Price.Prediction
2023.02.28 13:05:45.667 Storage Not authorized to open root of edit operation  (220000)
There is some kind of authorisation somewhere
 
Evgeny Dyuka #:
Yes, there is such a thing, but it gives an error
There's some sort of authorisation somewhere.

Well, I don't know. Try leaving and deleting the project, then rejoin it, then get an update from the repository.

When you leave a project in the log:

2023.02.28 16:31:47.481 Storage successfully left the project 'ONNX.Price.Prediction'

When deleting, nothing is written. When joining a project:

2023.02.28 16:31:59.970 Storage successfully joined to the project 'ONNX.Price.Prediction'

When receiving an update:

2023.02.28 16:41:43.985 Storage checkout started for MQL5\Shared Projects\ONNX.Price.Prediction
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\Python
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\Python\PricePredictionTraining.py
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\Python\PricePrediction.py
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\Python\model.onnx
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\ONNX.Price.Prediction.mq5
2023.02.28 16:41:44.750 Storage added MQL5\Shared Projects\ONNX.Price.Prediction\ONNX.Price.Prediction.mqproj
2023.02.28 16:41:44.784 Storage checkout completed successfully in 797 ms

Authorisation was only when connecting to the repository - it is impossible to join the project without it.

I can't help you any more. Unless you have an old version - I did it in the latest (beta and release).

 
Aleksey Nikolayev #:

Well, I don't know. Try leaving and deleting the project, then rejoin it, then get an update from the repository.

When you leave a project in the log:

When deleting, nothing is written. When joining a project:

When receiving an update:

Authorisation was only when connecting to the repository - without it it is basically impossible to join the project.

I can't help you with anything else. Unless you have an old version - I did it in the latest (beta and release).

got it, thanks
 
Evgeny Dyuka #:
Got it, thanks.

Just in case, I'm posting the project in the archive.

Files:
 
Renat Fatkhullin #:

What account did you log in to MQL5 Storage?

Did you specify your email or exact case-sensitive login?

Email is not accepted.

on login login, only my login is crooked initially, with dots dr.mr.mom, I have long ago, can you change the login?

 

Released beta 3589, on which you can test the operation of ONNX models.

Steps:

  1. Update to 3589 via Help -> Check Desktop Updates -> Latest Beta Version

  2. Activate MQL5 Storage by specifying the correct MQL5 login (not email) in case-sensitive form:






  3. If you have a corrupted config and MQL5 Storage does not work, open Explorer via File -> Open Data Folder, go to the MQL5 directory and delete the following files:


    . Then go to step 2.

  4. Join the ONNX.Price.Prediction project from the context menu using the Join command.



  5. Open Navigator, open the Shared Projects section and on the ONNX.Price.Prediction project call Update from Storage from the context menu:



    The project will be downloaded from MQL5 Storage (Subversion).

  6. Open the project and click on ONNX.Price.Prediction.mqproj - this is the project file



  7. Copy the project and run it on EURUSD, H1.

  8. Get the result:
    2023.02.28 21:54:43.316 Scripts script ONNX.Price.Prediction (EURUSD,H1) loaded successfully
    2023.02.28 21:54:43.348 ONNX    initialized [API version 1.14.0]
    
    в журнале Experts: важны две последние строки, а остальное - специально включенные отладочные логи по флагу ONNX_DEBUG_LOGS
    
    2023.02.28 22:11:53.441 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Creating and using per session threadpools since use_per_session_threads_ is true
    2023.02.28 22:11:53.441 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Dynamic block base set to 0
    2023.02.28 22:11:53.442 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Initializing session.
    2023.02.28 22:11:53.442 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Adding default CPU execution provider.
    2023.02.28 22:11:53.442 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Total shared scalar initializer count: 0
    2023.02.28 22:11:53.443 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Total fused reshape node count: 0
    2023.02.28 22:11:53.443 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Total shared scalar initializer count: 0
    2023.02.28 22:11:53.443 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Total fused reshape node count: 0
    2023.02.28 22:11:53.444 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Use DeviceBasedPartition as default
    2023.02.28 22:11:53.444 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Saving initialized tensors.
    2023.02.28 22:11:53.444 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Done saving initialized tensors
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Session successfully initialized.
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Number of streams: 1
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: Begin execution
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 0
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 1 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 1
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 2 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 3
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 6 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 4
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 8 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 5
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 9 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 6
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 7
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 2
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 3 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 11 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 8
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 9
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 16 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 10
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 11
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 22 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 12
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 25 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 13
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 27 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 27
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 17
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 18
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 28 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 28
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: ort value 30 released
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       ONNX: stream 0 launch kernel with idx 26
    
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       [-1.5921557]
    2023.02.28 22:11:53.445 ONNX.Price.Prediction (EURUSD,H1)       predicted 1.0591759612759748
    
  9. If it doesn't work, it means either an old version of ONNX Runtime or it is missing

    In this case, download the archive ONNX Runtime 1.14, signed by Microsoft, expand it to the root directory of the terminal next to terminal64.exe and restart the terminal:

In the project the simplest trained for the sake of example model.onnx is immediately attached so that you can see the demonstration.



To train the model yourself, you need to use the python scripts included in the project:

  1. Install Python 3 (e.g. Python 3.10) and deliver/update the packages:
    python.exe -m pip install --upgrade pip
    pip install --upgrade tensorflow
    pip install --upgrade pandas
    pip install --upgrade scikit-learn
    pip install --upgrade matplotlib
    pip install --upgrade tqdm
    pip install --upgrade metatrader5
    pip install --upgrade onnx==1.12
    pip install --upgrade tf2onnx
    The onnx package with explicitly state version 1.12 so as not to conflict with tensorflow requirements.

  2. Run MetaEditor and make sure that Python is recognised in Tools -> Options -> Compilers:



    If necessary, specify the path yourself, if the editor can't detect it by itself.

  3. The trading terminal should be connected to a server with EURUSD (for example, MetaQuotes-Demo)

    Make sure that integration with Python is enabled in the terminal settings. If it was switched off, switch it on and restart the terminal:




  4. Go to the ONNX.Price.Prediction project in the navigator and open the PricePredictionTraning.py file


  5. Run the script for compilation, which for a Python script means start.

    The script will connect to the terminal with the help of MetaTrader5 package (if it is switched off, it will start automatically) and start training on EURUSD, H1:


    When finished, the Journal editor will contain:
    .
    2023.02.28 22:24:16.876 Python  107/107 [==============================] - 0 s 1 ms/step - loss: 1.9167 - mae: 0.8553
    2023.02.28 22:24:16.876 Python  test_loss=1.917
    2023.02.28 22:24:16.876 Python  test_mae=0.855
    2023.02.28 22:24:17.179 Python  saved model to C:\Users\sys\AppData\Roaming\MetaQuotes\Terminal\8 D1E0A7047FFD62A8C00B09AEC1F3004\MQL5\Shared Projects\ONNX.Price.Prediction\Python\model.onnx
    2023.02.28 22:24:17.291 Storage modify MQL5\Shared Projects\ONNX.Price.Prediction\Python\model.onnx to base
    
    The new model.onnx will be saved/updated in the project. It can already be used.

  6. Run the PricePrediction.py script to make sure that the model works:
    .
    2023.02.28 22:27:46.078 Python  data path to load onnx model C:\Users\sys\AppData\Roaming\MetaQuotes\Terminal\8 D1E0A7047FFD62A8C00B09AEC1F3004\MQL5\Shared Projects\ONNX.Price.Prediction\Python\
    2023.02.28 22:27:46.078 Python  [[[1.061412 1.062328 1.060358 1.061169]]]
    2023.02.28 22:27:46.078 Python  [[[0.00105361 0.00119785 0.00074662 0.00125179]]]
    2023.02.28 22:27:46.078 Python  [[[ 0.18792516  0.17698306  0.01607251 -0.45455042]
    2023.02.28 22:27:46.078 Python    [-0.77068297 -0.50757406  0.2839477   0.36827372]
    2023.02.28 22:27:46.078 Python    [ 0.21639867 -0.3823502   0.48485409 -0.19891574]
    2023.02.28 22:27:46.078 Python    [-0.46696554 -0.31556414  0.33752274  0.48011389]
    2023.02.28 22:27:46.078 Python    [ 0.3302927   0.7196198   1.36884221  1.56656129]
    2023.02.28 22:27:46.078 Python    [ 1.63058293  1.46261473  1.39562973  1.74231014]
    2023.02.28 22:27:46.078 Python    [ 1.82040632  1.84663458 -0.26519644 -0.52644768]
    2023.02.28 22:27:46.078 Python    [-0.84661232 -0.50757406 -0.45270907 -0.86995678]
    2023.02.28 22:27:46.078 Python    [-1.27371495 -1.11699687 -1.10900328 -0.48650476]
    2023.02.28 22:27:46.078 Python    [-0.82762998 -1.37579285 -2.0599602  -1.62088366]]]
    2023.02.28 22:27:46.094 Python  [[-1.6349795]]
    2023.02.28 22:27:46.094 Python  predict:  [1.05912]
    
  7. The test example worked!


This is a specially created example to demonstrate both training and running the models.

You can convert your own TensorFlow models to ONNX format using the python packages onnx and tf2onnx:

import tf2onnx
...
onnx_model = tf2onnx.convert.from_keras(model, output_path=output_path)

ps: use the latest version of the project, as the function prototypes have changed in the old version.
 

It worked.

2023.03.01 06:54:34.987 ONNX.Price.Prediction (EURUSD,H1)       [-0.20001383]
2023.03.01 06:54:34.988 ONNX.Price.Prediction (EURUSD,H1)       predicted 1.0581684510582798
 
Renat Fatkhullin #:
  1. Put in Python 3 (e.g. Python 3.10) and deliver/update packages:
    Package onnx with an explicit mention of version 1.12 so as not to conflict with tensorflow requirements.

Could not install this package. First it required CMake to be installed. Now it requires Visual Studio to be installed?

CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
 
Aleksey Nikolayev #:

Could not install this package. First they demanded to install CMake. And now they require Visual Studio to be installed?

You are still at the beginning of the road of fighting the hell of mutual incompatibilities and creating separate environments for tasks in Python.

Some packages require compilers for the target build.

 
Renat Fatkhullin #:

You're the one who is still at the beginning of the journey of dealing with the hell of mutual incompatibilities and creating separate environments for tasks in Python.

Some packages require compilers for the target build.

Absolutely agree! It's a lot of work. Two questions in this regard:

1. Can models created using GPU be converted to ONNX?

2. We don't need Python to get the model prediction in ONNX in the MQL5 Expert Advisor, do we? The calculations will be performed by the onnxruntime environment?