MT5 tester bug: same conditions, different results

 

Hi all!

I found a few similar posts (questions) with no solution: when run the same backtest two times (one after another) , the MT5 tester sometimes gives different results.
The conditions (real ticks, time range, etc.) are the same.


I run the backtests on my laptop. The latest stable build of MT5 terminal.

Can anyone help with this?

 
  1. Look and compare single trades to find the reason for the difference.
  2. Don't assume that backtests reflect reality, they are maximally similar.
 
Carl Schreiber #:
  1. Look and compare single trades to find the reason for the difference.
  2. Don't assume that backtests reflect reality, they are maximally similar.

Hi Carl,

As I stated in the initial request, all the conditions are the same. It's just MT5 tester that gives sometimes different results with the same input.

 
Alexander Kalinkin #:

Hi Carl,

As I stated in the initial request, all the conditions are the same. It's just MT5 tester that gives sometimes different results with the same input.

I made an experiment now - I backtested one EA two times with same input.
EA is uploaded on this post (by source code).
It is the condition (EA with default settings):



---------------------------

First backtesting:

Second backtesting:

Files:
backtest1.zip  63 kb
backtest2.zip  62 kb
 
Alexander Kalinkin #:

Hi Carl,

As I stated in the initial request, all the conditions are the same. It's just MT5 tester that gives sometimes different results with the same input.

That just mean the "conditions" are not the same, or your EA contains some "randomness".

There is no MT5 bug unless your prove it otherwise by providing all the information, parameters and code to reproduce it.

 

So, we can see about why those two backtesting results are slightly different (one from an other):
the number of ticks are different.

  • It is not a bug. It is because the Metatrader is automatically loading the ticks which strategy tester needs for backtesting.
  • Besides, it is not a weekend today. Today is Monday and forex market is open.
  • And it also depends on the quality of the data/datafeed of the broker
    (because backtesting with "every tick based on real ticks" is backtesting with the data provided by the broker).

It is not big problem for EAs which using classical trading systems, but it may be the problem for martingale and tick scalper EA, EA using open bar and close bar and high/low of open bar all together on the way as well-known Firebird EA, and some more.

--------------------

It is not a bug.

 
Sergey Golubev #:

So, we can see about why those two backtesting results are slightly different (one from an other):
the number of ticks are different.

  • It is not a bug. It is because the Metatrader is automatically loading the ticks which strategy tester needs for backtesting.
  • Besides, it is not a weekend today. Today is Monday and forex market is open.
  • And it also depends on the quality of the data/datafeed of the broker
    (because backtesting with "every tick based on real ticks" is backtesting with the data provided by the broker).

It is not big problem for EAs which using classical trading systems, but it may be the problem for martingale and tick scalper EA, EA using open bar and close bar and high/low of open bar all together on the way as well-known Firebird EA, and some more.

--------------------

It is not a bug.

Hi Sergey,

Yes, my EA works on every tick. However, I use time range with latest date in the past (e.g. 01.01.2016-05.05.2023) so the MT should not add automatically new ticks to the testing volume.

Please find below the example: I ran optimization with change in only magic number. All the results but one are the same, one is wrong.

sett

opt_res


What's wrong here? Why one result is different? It's only magic number, nothing more.
If I re-run this one (double-click and run the single test) this will show normal result as others (958 trades). But here in optimization results it is obviously wrong.

 
Alexander Kalinkin #:
Yes, my EA works on every tick. However, I use time range with latest date in the past (e.g. 01.01.2016-05.05.2023) so the MT should not add automatically new ticks to the testing volume.

From 2016 to 2023?
I think - some ticks had been adding to strategy tester from Metatrader.

Because if I backtested simple (simple means: no tickscalping EA, no martingale, no pyramiding, and so on) from
2 of June till 12 of June (this year), and Metatrader added more than 20K ticks for few minutes to strategy tester after first backtest so
the ticks should be added in case of you are backtesting from 2016 ...

I did not check the optimization now but I think ... if you are repeating same optimization (with same condition) many times so
strategy tester can use the previous optimization results to speed the process up so
"too much optimization is not good" (especially for the EAs working with ticks such as TickSpalper EA, PolyfitScalper EA and more).

--------------------

Besides, it also depends on EA.
Do you remember old Firebird EA? (it was uploaded to some very old threads here by source codes with many versions, with trading results and with the settings proposed).
This EA was coded to work on open bar together with high/low of open bar and close/previous bar as well (all together) - M15 as I remember.
Yes, it was MT4 ... and the backtesting results were different and depended oon the qualuty of the history/datafeed and when we start backtesting and more and more (and which broker for example).

--------------------

So, if the next (same but next) backtesting result is different from the previous one so it means the following:
the condition for this EA were not same one (not for you; it may be same condition for you but not same for the EA).

 
Sergey Golubev #:

From 2016 to 2023?
I think - some ticks had been adding to strategy tester from Metatrader.

Because if I backtested simple (simple means: no tickscalping EA, no martingale, no pyramiding, and so on) from
2 of June till 12 of June (this year), and Metatrader added more than 20K ticks for few minutes to strategy tester after first backtest so
the ticks should be added in case of you are backtesting from 2016 ...

I did not check the optimization now but I think ... if you are repeating same optimization (with same condition) many times so
strategy tester can use the previous optimization results to speed the process up so
"too much optimization is not good" (especially for the EAs working with ticks such as TickSpalper EA, PolyfitScalper EA and more).

--------------------

Besides, it also depends on EA.
Do you remember old Firebird EA? (it was uploaded to some very old threads here by source codes with many versions, with trading results and with the settings proposed).
This EA was coded to work on open bar together with high/low of open bar and close/previous bar as well (all together) - M15 as I remember.
Yes, it was MT4 ... and the backtesting results were different and depended oon the qualuty of the history/datafeed and when we start backtesting and more and more (and which broker for example).

--------------------

So, if the next (same but next) backtesting result is different from the previous one so it means the following:
the condition for this EA were not same one (not for you; it may be same condition for you but not same for the EA).

Hi! 
My backtest has fix time period in the past (as it's seen from the screenshot) so no new ticks should be added if  the range is unchanged. 
The idea that the tester uses previous optimization results to speed up the process.. It should not do so because slow complete optimization assumes that you want to see all the results... But I may be wrong in my reasoning though. Is there a parameter to additionally force the tester to run the backtest without looking for previous results? 
 
Alexander Kalinkin #:
Hi! 
My backtest has fix time period in the past (as it's seen from the screenshot) so no new ticks should be added if  the range is unchanged. 
The idea that the tester uses previous optimization results to speed up the process.. It should not do so because slow complete optimization assumes that you want to see all the results... But I may be wrong in my reasoning though. Is there a parameter to additionally force the tester to run the backtest without looking for previous results? 
I just explained everything in theoretical points of view according to my experience (as the ideas for example).

I think it may be a lot of particularities, and I am using backtesting to be sure that EA is working and can open the trades,
about how stop loss and trailing stop is working, and to have some ideas that EA may be profitable in some cases with some brokers and so on.

I know that there are the people who use backtesting result as a proof that EA is profitable ... but I am not one of them ...
 
Sergey Golubev #:
I just explained everything in theoretical points of view according to my experience (as the ideas for example).

I think it may be a lot of particularities, and I am using backtesting to be sure that EA is working and can open the trades,
about how stop loss and trailing stop is working, and to have some ideas that EA may be profitable in some cases with some brokers and so on.

I know that there are the people who use backtesting result as a proof that EA is profitable ... but I am not one of them ...

Got it, thanks!
Will continue to search for an explanation.