Slow backtest and historical data

New comment
 
Hi, I ask the more experienced a doubt since I have not been involved in MQL4 programming for long. Up to now I have used Alpari data at 90% for backtests, but seeing that everyone recommends doing them at 99% (Since I understood, the difference is considerable) I tried to download the data from the internet with the various programs available .
In the end I had to install the MT4 in portable version (I found the guides here on the forum and everything went well and it is working) and I downloaded everything including the historical data on an external hard drive because my internal does not have enough space.
The question is this: it really takes a long time to do a backtest compared to before, let's talk that it took 2 hours longer for the same expert. Is this normal and may it depend on my code or is it because I am using an external hard drive which may not be fast enough?
 

Yes, using real tick data is much slower, but reason depends on both issues. Your code is usually the one with the most impact on the testing speed.

The usual problems are due to code doing things on every tick instead of only when a new bar forms.

The more efficient your code the better.

The external harddrive issue also has an impact obviously. So using an internal drive or an SSD will definitely help improve the speed of tests.

 
Fernando Carreiro # :

Yes, using real tick data is much slower, but reason depends on both issues. Your code is usually the one with the most impact on the testing speed.

The usual problems are due to code doing things on every tick instead of only when a new bar forms.

The more efficient your code the better.

The external harddrive issue also has an impact obviously. So using an internal drive or an SSD will definitely help improve the speed of tests.

Thank you so much for the quick and kind reply.
Actually something happened to me that I found strange but it certainly depends on what you told me: with the historical data at 90% I had not inserted any function of this type that operates only at the opening of a new bar instead of every tick, but everything worked normally.
When I tried with this data at 99% instead I got the error array out of range. I inserted the function to execute only once when the bar was opened and everything was working again.
How can this depend on the historical data used?
For the rest in addition to the code I understood, I imagined it could also center the hd , surely I have to fall back on a better product.
 
forextastic #:
Thank you so much for the quick and kind reply.
Actually something happened to me that I found strange but it certainly depends on what you told me: with the historical data at 90% I had not inserted any function of this type that operates only at the opening of a new bar instead of every tick, but everything worked normally.
When I tried with this data at 99% instead I got the error array out of range. I inserted the function to execute only once when the bar was opened and everything was working again.
How can this depend on the historical data used?
For the rest in addition to the code I understood, I imagined it could also center the hd , surely I have to fall back on a better product.

If you are getting array out of range then your code is faulty. It is probably not checking the number of available bars or something like that.

 
Fernando Carreiro # :

If you are getting array out of range then your code is faulty. It is probably not checking the number of available bars or something like that.

Ok but how is it possible that this happens on historical data at 99% and instead works without error at 90%? Shouldn't an error appear either way?
I ask to understand, not because I think there is no mistake on code.
 
forextastic #: Ok but how is it possible that this happens on historical data at 99% and instead works without error at 90%? Shouldn't an error appear either way? I ask to understand, not because I think there is no mistake on code.

Because FXT files generated by the terminal always generates extra 1000 bars before the test start date, but when you use 3rd party tools for generating FXT files with tick data, you the user may forget about this necessity and probably only generate data from the start date you plan to use.

As I said, the problem is your code, because you are probably not doing proper checks into indexing of bars and indicators. No matter the reason, you need to fix you code anyway.

 
Fernando Carreiro # :

Because FXT files generated by the terminal always generates extra 1000 bars before the test start date, but when you use 3rd party tools for generating FXT files with tick data, you the user may forget about this necessity and probably only generate data from the start date you plan to use.

As I said, the problem is your code, because you are probably not doing proper checks into indexing of bars and indicators. No matter the reason, you need to fix you code anyway.

Of course, I have already fixed the problem (as mentioned it was enough to insert the calculation instead of on each tick, only at the opening of the bar).
I was amazed by the difference in the result between 90 and 99% in terms of backtesting.
I am now equipping myself with an SSD to improve speed. Thanks for your suggestions
 
Fernando Carreiro # :

Yes, using real tick data is much slower, but reason depends on both issues. Your code is usually the one with the most impact on the testing speed.

The usual problems are due to code doing things on every tick instead of only when a new bar forms.

The more efficient your code the better.

The external harddrive issue also has an impact obviously. So using an internal drive or an SSD will definitely help improve the speed of tests.

Now I really understand what you meant and thanks for the advice. I bought an SSD and it actually took me no longer a day but 4 hours for a backtest. I searched for information, worked on code and optimization, and was able to take the backtest from 4 hours to just over two minutes.
Now I understand that the optimization of the code is very important and must be written immediately in an appropriate way!
 
forextastic #: Now I really understand what you meant and thanks for the advice. I bought an SSD and it actually took me no longer a day but 4 hours for a backtest. I searched for information, worked on code and optimization, and was able to take the backtest from 4 hours to just over two minutes. Now I understand that the optimization of the code is very important and must be written immediately in an appropriate way!

It is very good that you have understood that so quickly. Very few have discovered that. Well done!

 
Fernando Carreiro # :

It is very good that you have understood that so quickly. Very few have discovered that. Well done!

But it is normal that I tried to create an expert advisor that breaks out of the daily High and Low levels (it is an example), and while everything works perfectly on Alpari with historical data at 90% (I also inserted a Comment to see the prices and everything goes smoothly) on the historical data at 99%, the expert does not work as it should? That is, the comment does not give me the prices and does not enter the market as it should.
However, this only happens if I try to take the prices from another timeframe, if I do a test with expert advisors who do not have this particularity, everything seems to be fine ( I guess ).

 
forextastic #: But it is normal that I tried to create an expert advisor that breaks out of the daily High and Low levels (it is an example), and while everything works perfectly on Alpari with historical data at 90% (I also inserted a Comment to see the prices and everything goes smoothly) on the historical data at 99%, the expert does not work as it should? That is, the comment does not give me the prices and does not enter the market as it should.However, this only happens if I try to take the prices from another timeframe, if I do a test with expert advisors who do not have this particularity, everything seems to be fine ( I guess ).

When your EA requires data from a different time-frame or a different symbol, you have to carry out extra operations to make sure that the data has been retrieved and synchronised properly. The data is not explicitly available for you the moment the tester begins. You have to check for delays and errors.

Forum on trading, automated trading systems and testing trading strategies

Retrieving current history data from symbol that doesn't have an open chart on a particular period

Fernando Carreiro, 2022.04.25 11:20

In Multi-Symbol/Multi-Time-frame Indicators and/or EAs, you have to properly handle the synchronisation of the requested of data on a timely fashion by handling 4066/4073 errors. Failure to do this will cause it not to work properly.

Forum on trading, automated trading systems and testing trading strategies

Minimum requirements for showing different symbol info in-time info in one chart?

William Roeder, 2021.08.27 17:06

On MT4: Unless the current chart is that specific symbol(s)/TF(s) referenced, you must handle 4066/4073 errors before accessing candle/indicator values.
          Download history in MQL4 EA - Forex Calendar - MQL4 programming forum - Page 3 #26.4 (2019.05.20)

On MT5: Unless the current chart is that specific pair/TF, you must synchronize the terminal Data from the Server before accessing candle/indicator values.
          Error 4806 while using CopyBuffer() - Expert Advisors and Automated Trading - MQL5 programming forum #10 (2020.12.15)
          Is it mystical?! It is! - Withdraw - Technical Indicators - MQL5 programming forum (2019.05.31)
          Timeseries and Indicators Access / Data Access - Reference on algorithmic/automated trading language for MetaTrader 5
          Synchronize Server Data with Terminal Data - Symbols - General - MQL5 programming forum #2 (2018.07.17)
          SymbolInfoInteger doesn't work - Symbols - General - MQL5 programming forum (2019.09.03)

12
New comment