Different results from different brokers MT platform while using strategy tester

New comment
 

Hi,

I'm trying to test an expert in strategy tester in the same computer on two different brokers MT4 platform. They are both build 950.

I have downloaded M1 historical data from FXDD and by using period converter, convert it up to Monthly data sucessfully in both terminals, checked all TFs, fine. Report gives 90% tick quality both.

The expert has same inputs, same data, same slippage in both terminals however results are slightly different. If it would be a demo or live, I would consider the difference as normal however both test are performed in strategy tester. No latency issue, no hidden slipage or different spreads etc etc.

I know that brokers could develop some addition things at the top of MT4 terminal but I wonder if they could able to implement something that would change the expert behaviour in strategy tester level?

How can I purely backtest the expert behaviour off-line without any broker impact?

Thanks

 

That is to be expected, as the current or week-end closing prices of the currency pairs will be slightly different on various brokers. This is noticeable when the currency pair being tested requires an exchange rate to be applied to be calculated for the Deposit/Balance currency.

For example, lets say you are testing the "GBP/JPY" but your Deposit/Balance currency is in "USD". In this case your Profit or Loss has to be converted from "JPY" to "USD" and to do so the Strategy Tester uses the current exchange rate given by the broker which can be slightly different on another broker.

However, if you were testing the "EUR/USD" and your Deposit/Balance is "USD", then there should be no difference between brokers as the tick value is already in "USD" and so no exchange rate need be applied (except for other factors).

EDIT: You can also read the following post as it is related: https://www.mql5.com/en/forum/159037

 

Hi FMIC,

Thanks for your prompt answer. Since the data is the same (downloaded from as M1 data) I'm not sure if it's related with weekend closing prices. I 'm testing 2015-Q3 to 2016-Q1, so no recent data tested.

The deposit issue was something I did not consider however the # of open positions changed as well. In one broker it opens 30 positions, other 31. I suspect from data but the data is completely same. I've fully clear all M1 to Monthly data from history in both cases and import the M1 data fresh. Both account is USD based and I'm testing GBP/USD.

I've checked the topic you mentioned, in my case, every time I run the code in the same broker, it gives the same result. There is no change in the same broker or in the same terminal. So there can't be any randomless issue in the code.

 

aed71:

Since the data is the same (downloaded from as M1 data) I'm not sure if it's related with weekend closing prices. I 'm testing 2015-Q3 to 2016-Q1, so no recent data tested.

The deposit issue was something I did not consider however the # of open positions changed as well. In one broker it opens 30 positions, other 31. I suspect from data but the data is completely same. I've fully clear all M1 to Monthly data from history in both cases and import the M1 data fresh. Both account is USD based and I'm testing GBP/USD.

Even if there is no currency conversion on testing GBP/USD with USD Balance, there can still be differences due to specific broker conditions (for example different values between brokers such as "Leverage", "Stop Level", "Freeze Level", "Lot Size: Min,Max & Step" and others).

Also, even though you downloaded the same M1 data, there is no guarantee that the "emulated" tick data that MetaTrader produces will be exactly the same every time on different setups (due to minor factors in play).

The only way to guarantee exact match is to have an exact same FXT file produced with "real" tick data (and the same properties/conditions), and to do that you will need to use 3rd party tools such as "TickStory" or "Birt's Tick Data Suite" (or others).

The following screenshot from TickStory shows the properties ("Leverage", "Stop Level", etc.) that are stored in a FXT file and that affect the outcome of the back-test (and thus cause differences between two different setups):

FXT Header

EDIT: PS! Also remember that different brokers may also have different Time-zones that will also affect the outcome.

 

One more item I need to mention and it is a very important one.

Obviously I understand that you want to make sure that your EA is running correctly and that you are executing your back-tests properly and that is why you are trying to identify the differences when testing your EA against various brokers.

However, it is actually a GOOD thing that you are getting differences as that will allow you to test your EA against various environments and brokers and thus improve its functionality so as to adapt to those differences.

 

Thanks again, you are giving valuable information, I'll check tickstory. You are also right that I'm testing in different environment which sould make me more comfortablebut in fact I'm not  :-))

The data I downloaded is not 99% quality, it's 90% and this makes difference between real trading and backtesting (even 99% could make difference, I accept this). Which I couldn't understand is that the terminal softwares have the same build thus should excatly behave same during strategy testing, independent of broker's values/setting etc. I'm not testing on demo or live, this could make difference but backtesting should not. I could also accept that backtest data I've downloaded is not accurate and in demo or live trading results could make differences but in the same dataset it should behave exactly same.

Suppose that I want to sell or hire my expert. I should show some backtest results, calculate sharpe ratio etc, should I say, according to Broker-A's terminal backtest results profit is 5263USD and according to Broker-B's terminal it is 4712USD? My backtest results are as below:

Broker-A:

Profit 5263USD, profit factor=3.06 Trades=33 Drawdown 8.20%

Broker-B:

Profit 4712USD, profit factor 2.95 Trades=32 Drawdown  6.33%

If I cannot find and accept a reason why two terminal results are different, I would conculde that the additions that brokers made on MT4 terminal could impact expert trading! They could as well stop/block some functionality in the expert with purpose or without purpose.

PS.Previously I observe a different issue between a broker's demo system and live. I was running an expert that calculates linear regression slope. I have tested and verify in the same brokers terminal with demo account and when I connect to live from the same terminal the linear regression didn't work. The same broker, same expert (even in the same directory with same settings) but probably different servers made difference. It was only drawing the regression no trading. The regression values became zero in live but working perfectly in demo. Broker didn't find anything or didn't really checked!

 
aed71:

Thanks again, you are giving valuable information  ...

  1. As I explained, MetaTrader Strategy Tester produces "emulated" tick data, not real tick data. You have to use external tick data and use 3rd party tools to be able to get 99% quality.
  2. Even with 99% quality, brokers are different and apply different rules and conditions resulting in different outcomes (Leverage, Stop Level, Spread, Commission, Lots, Swaps, etc.).
  3. Even if a EA does very well in a back-test, there is no guarantee that it will do well in a live forward test, because conditions change (Market, News Events, Sentiment, etc.).
  4. Also, very common, is the fact that many issues with EAs not producing expected results, are actually caused by bugs and lack of knowledge and experience by the coder.
    (Even though I do not know how your Linear Regression EA works, I can tell from your explanation, that it is not a broker problem but an EA issue or bug or lack of understanding about the trading conditions).
PS. Due to forum rules, there are certain things that I am not allowed to divulge here, so if you need private help, contact me via a PM!
 

I've just finilized another fresh installation of MT4 downloded from metaquotes.net, load the same history data (M1 to Monthly) and copy my indicators/expert.

The same testing performed between same dates and same spread etc.

Broker-C:

Profit 3854USD, profit factor=2.91 Trades=25 Drawdown 6.19%

Between broker-A and broker-C there is a huge difference in terms of profit and # of trades.


I also did a different test, I've copied the ex4 expert of broker-a terminal into broker-b terminal (not perform compile) the results still different.

I suspect from the implementing of historical data, but will try to verify each data is copied as same in TF 5 and above. I'm sure about M1 data but how period converter works I'm not sure about it.

 
FMIC:
  1. As I explained, MetaTrader Strategy Tester produces "emulated" tick data, not real tick data. You have to use external tick data and use 3rd party tools to be able to get 99% quality.
  2. Even with 99% quality, brokers are different and apply different rules and conditions resulting in different outcomes (Leverage, Stop Level, Spread, Commission, Lots, Swaps, etc.).
  3. Even if a EA does very well in a back-test, there is no guarantee that it will do well in a live forward test, because conditions change (Market, News Events, Sentiment, etc.).
  4. Also, very common, is the fact that many issues with EAs not producing expected results, are actually caused by bugs and lack of knowledge and experience by the coder.
    (Even though I do not know how your Linear Regression EA works, I can tell from your explanation, that it is not a broker problem but an EA issue or bug or lack of understanding about the trading conditions).
PS. Due to forum rules, there are certain things that I am not allowed to divulge here, so if you need private help, contact me via a PM!


I know that broker's apply different rules and conditions but backtest should not vary from broker to broker. It could differ from histroical data to historical data but not from broker to broker.

I know that my expert will not work in live as in backtest due to sliappage, market conditions, different spread, different LPs etc but I would expect that it gives same results with same settings and data in backtesting. I currently could not get a "reference" performance of my expert. The live performance is something on top and different issue.

The linear regession case, it works in other broker's live and demo but only in one broker it works in demo but not live. I'm pretty sure that it's not a coding issue since the same ex4 works in different brokers. I also think that my case is not a coding issue, I run it many times with the same settings, getting same results but when I change the terminal it differs. I'll make a couple of different setups and try to undersand if it's a historical data implementation issue or not. Thanks.

 

aed71: >> I know that broker's apply different rules and conditions but backtest should not vary from broker to broker.

Wrong! I have already pointed out (several posts) that there are many reasons for being different and I have already listed several, yet you have not comment once on them.

  • Do you know what "Stop Level" or "Freeze Level" is? Does you code adjust stop sizes based on it?
  • Does you EA adjust the prices based on the tick size (and not the point size which is the incorrect method)?
  • Does you code check for the minimum, maximum or step size of lots?
  • Does you code check the leverage and margin risk?
  • etc.

aed71: >> but I would expect that it gives same results with same settings and data in backtesting.

Wrong again! I have already pointed out that the Strategy Tester "emulates" tick data and that is NOT the same as "real tick data", nor does it have delays or latency problems or re-quotes that live or demo forward tests have.

aed71: >> I'm pretty sure that it's not a coding issue since the same ex4 works in different brokers.

Wrong again! You have just described the classical "bug" - your code is unable to adapt to a particular condition of which you are not aware. In the very least your code should detect the abnormality and warn you about it. The very fact that you don't know WHY it is happening with just that broker's live account is the proof that it is a "condition" that the code (and you) does not cater for. There is an old saying in programming - G.I.G.O; which is quite cynical, but roughly translates into the idea that a program can only be as good as the knowledge that went into it, and when it fails it is in fact the coder/programmer that failed and that the "bug" is in the human (not the machine). That is why coders/programmers have to continually improve their knowledge and experience in order to improve their code.

EDIT: PS! If you want, send me a PM and the details of the broker account that does not work and I can test your ".ex4" file on it and offer you debugging advise.

 
I've already answered all your questions but your are trying to pointing the issue to the code ONLY.

I'm not able to send my code even ex4. The example I gave about the linear regression code was 2 years ago. I'm not even using it and the issue is not that code, it was just an example that there can be different implementations of brokers even in their different servers.

The issue is backtesting (even offline) differences between MT4 terminals. The issue is also not real trading vs backtesting differences. Why the same expert works differently in broker-a and broker-b terminals.

I'm currently suspecting historical data differences between terminals. I need to check the consistancy of historical data between terminals.

12
New comment