MT5 Data - page 2

 

Hi lori,

 there are lots of problems with data, as well as a number of bugs in the MT5, but i am not sure what will you resolve by moving to another platform (but i would love to know what options you are considering).

 I have done a lot of investigation regarding data and quality of data since MT4 , and IMHO, the only way to ensure that results of your backtests are 'correct' is to know what problems there are and how to bypass them.

As i see it, the main problem is that FX is 'unregulated' market, so every broker has a different values for same time (after timezone correction). Even the same broker, gives different prices to different terminals (i have  two MT4 terminals on different computers on different internet connection on SAME REAL account, and when i compare the values, there are differences, up to 3 pips difference, up to 50 M1 values daily).

So there is no hope that you can have a 'true' data,  or correct by any criteria: the best data on which you can do the backtests is the one you have collected by being connected all time to your broker, as it also differs from the data your broker will give you later: i.e. you make a fresh install and what you get is different from what you have on old terminal. i cannot see reason for that, but it is so (MT4, i don't have real MT5 account yet).

It is incredible how much the results of an EA can differ based on the data you are adjusting your EA to (i am talking about same instrument, but different providers). So the problem is actually how to adjust the behaviour of your EA to be as robust as possible. How to make it  parameter independent as much as possible. And if you start looking at the issue in this way, you will get to complete new set of problems.

 

MT5.
As you might have noticed, there are two sets of data for the same instrument in MT5. One is in the trading terminal, and the other is in strategy tester.

In trading terminal, you can see M1 for aprox last 3 months, and this is probably your brokers consolidated data.

In strategy tester, i am not sure how much you can get (depends on broker, settings etc.) but it is much more. The most important thing is that this data IS THE SAME ACROSS COMPLETE AGENT NETWORK, so there is a complete new dimension in population size backtesting possibility. The question is, how this data is compiled, and how different it is from your brokers data and how is this affecting your backtest precision. So we are coming again to same question, how to make a robust system which can compensate for this?

The problem you are talking about,  data repeating,  is superficial: the true problem is how to calculate the 'best' value from all inputs you are receiving, and what to do with missing time zones?
Because if you want to develop a non-parametric system, truly robust one, the data has to be continuous,  otherwise you get undefined zones: e.g. if you use derivatives to determine turning point/local minimum of price movement, data must exist across whole time domain.

 

etc.

 
lori:

Several brokers had more than 1 set of OLHC for the same period of time! This is absolutely crazy - I really cannot believe that MetaQuotes would not use some kind of sanity check on its data to ensure it follows a certain set of criteria.

Let me illustrate with one simple point in time:

$ grep '2010.01.04 11:57' Files/*
Files/4367984-GBPUSD-M1-AlpariFS-MT5.csv:2010.01.04 11:57,1.61361000,1.61360000,1.61379000,1.61377000
Files/4367984-GBPUSD-M1-AlpariFS-MT5.csv:2010.01.04 11:57,1.61750000,1.61746000,1.61778000,1.61769000
Files/4477443-GBPUSD-M1-MIGBank-Demo.csv:2010.01.04 11:57,1.61751000,1.61751000,1.61778000,1.61770000
Files/4477443-GBPUSD-M1-MIGBank-Demo.csv:2010.01.04 11:57,1.60664000,1.60577000,1.60678000,1.60618000
Files/4477443-GBPUSD-M1-MIGBank-Demo.csv:2010.01.04 11:57,1.61751000,1.61751000,1.61778000,1.61770000
Files/4676701-GBPUSD-M1-InstaForex-Server.csv:2010.01.04 11:57,1.61750000,1.61750000,1.61780000,1.61770000
Files/4745042-GBPUSD-M1-MetaQuotes-Demo.csv:2010.01.04 11:57,1.61751000,1.61751000,1.61778000,1.61770000

Wow - AlpariFS has 2 entries for 4th January 2010 whilst MIGBank has 3 entries!

If I can jump to conclusions here I suspect what has happened is that both AlpariFS and MIGBank have loaded the MetaQuotes-Demo historical data on top of theirs (you can see that 1 of AlpariFS's ticks and 2 of MIGBank's ticks match the MetaQuotes-Demo tick).

The fact that MetaTrader allows duplicate dates is so dodgy - and it really calls into question the reliability of their system. It is for this reason that I have decided to move to another platform.

I hope this thread helps others others evaluate the quality of the back testing available in MetaTrader5.

I have checked your statement for the Alpari-FS, it didn't reproduce


 
Rosh:

I have checked your statement for the Alpari-FS, it didn't reproduce


Rosh - I agree - I have rerun my script and also get unique data now - can I claim credit for my previous rant making them clean up their data? :)
 

graziani:

... snip ... 

I have done a lot of investigation regarding data and quality of data since MT4 , and IMHO, the only way to ensure that results of your backtests are 'correct' is to know what problems there are and how to bypass them.

... snip ...

The problem you are talking about,  data repeating,  is superficial: the true problem is how to calculate the 'best' value from all inputs you are receiving, and what to do with missing time zones?

Because if you want to develop a non-parametric system, truly robust one, the data has to be continuous,  otherwise you get undefined zones: e.g. if you use derivatives to determine turning point/local minimum of price movement, data must exist across whole time domain.  

Hi Graziani

Thanks for your input - I understand what you're saying, and yes, perhaps I have been hasty in dismissing MT5 outright. 

So assume that we accept these issues - that in fact there will be gaps in the data when doing back testing. How does one account for this? If you can't rely on the back test how can you be confident the strategy will work?

As for which other platform I was thinking of - I have heard glowing reports about TradeStation - I haven't looked into them much though. Their market data page is pretty impressive. I hear their data quality is very high.

Streaming Market Data for Online Trading | Trade Online with TradeStation
  • www.tradestation.com
TradeStation's unmatched degree of speed and reliability of data transmission ensure that you know where the market is before others do. Avoid back-testing and trading pitfalls caused by bad ticks TradeStation's data is among the cleanest in the industry, allowing you to make your trading decisions with confidence. TradeStation understands the...
 

Well ... I will help the staff here. If you want to download quality data, I recommend the TrueFX. In my opinion is the best so far, but the historical data can be great after that turn to HST.

 
tcferreira:

Well ... I will help the staff here. If you want to download quality data, I recommend the TrueFX. In my opinion is the best so far, but the historical data can be great after that turn to HST.

Sadly MT5 does not support .hst files