Collected tick data graph vs. original graph? Why the difference?

 

I've started to collect tick data lately for testing purposes (that have nothing to do with this thread). Just out of curiosity I used simple csv2fxt (https://www.mql5.com/en/code/8658) script to convert the data into an hst file. I took a spare MT4 installation, deleted the hst files and replaced them with mine. I opened a chart based on my collected data and was shocked by the differences between this graph and the normal one in my original MT4 installation (the one used to collect the data from a Live server).


Now these ticks where collected with the most minimalistic approach possible. Just a single function in start() that writes TimeCurrent() and Bid to CSV file. I find it very hard to believe that there could be anything wrong with it.


The differences where very easily visible on each and every bar. Some spikes where actually completely absent from my graph. How can this be? I thought MT4 builds the graph based on ticks coming in. If I capture all ticks to file, I should be able to reconstruct the graph 1 to 1. Am I missing something? I am almost panicking here... Help!

 

Hi Gordon

There could be something wrong with the conversion process. The graphs you are looking at are on a 1 minute timescale I assume and the conversion process might be putting the ticks in each minute candle slightly out of phase that would completely change the look of the graph. A spike may occur at the begining or end of a minute candle but the conversion process puts it in the next or previous candle if you see what I mean. People forget that computers are sequential and although the current time when read is correct there is a delay by the time it reads the price especially if it has to go to the server to get it so it is highly possible for a tick to be recorded at a different time by a few milliseconds or more. This is why people don't use tick data to trade only longer timescale candles.

 

Well, the conversion was done with a very well reputed script (https://www.mql5.com/en/code/8658) by stringo and it seems to work flawlessly. I have manually checked some of the candle sticks produced in excel and got the same result. I doubt there is a problem with the script, but I guess I'll check the code anyway.

Regarding the delays you are talking about - I agree there are delays but its irrelevant cause ticks come in 'attached' to a server time. When u reaad TimeCurrent() you r actually reading the last known server time and NOT the current time. The last known server time comes WITH the last quote (last tick). So when if u RefreshRate() and then read Bid and TimeCurrent() the time is supposed to fit the quote regardless of delays. Anyway, if delays would affect that then in every part of the world people would be looking at different charts... But the charts are the same. There is a delay in chart update for the last few hundred mili-seconds (maybe even seconds if your connection is really bad), but when u get the data it's supposed to be the same.

Does anybody have technical knowledge of how the MT4 graph is being updated in real time? I am guessing at the moment that regardless of the ticks u receive,the graph is updated with high/low/open/close at the timeframe u r viewing once every timeframe period independently than the ticks u receive. If that's true then the graph does not really represent the prices that where available to you for trading. At the end of the day I dont really care whats on the graph if my expert could only c some of the prices available on the graph in real time. And I know that I wont see each and every price in every candle, but the minimum I should have gotten was open/close/high/low of each candle.

 
Common sense should tell you if the data is correct it can only be the alignment of the data with the candles and the conversion process is responsible for the alignment. If you put the same data in a history file you will get the same chart every time so clearly the processing of the data is the problem.
 
I have looked it through. There are many cases where a local high or low that appears in the original graph is nowhere in the new graph. Even if there was miss alignment it should be there.
 
You have proved the collection of data is the problem it doesn't contain all the ticks. What do you think happens to simultaneous ticks or ticks very close together within 1 mS. The server must hold them in its history file for later download but what does it send in real time. There has to be a compromise since it can't get the realtime data across the internet. There are big differences between brokers as well some give very few ticks and others are very spikey. Over bigger candle times that most people use for trading ticks become irrelevant.
 
So from what u r saying I understand that the graph represents ALL the price quotes at THE BROKER'S SERVER, but an expert adviser only gets SOME of those price quotes (because of technical reasons like u said)? Sounds like a ripoff to me. If that's true then any broker can avoid sending certain quotes and blaim it on technical problems. Whats the point of having a graph that DOES NOT represent the actual quotes u get.
 

All the graphs show an average value of the price. Compare different brokers to see the differences in the charts you will be amazed. The brokers also get up to all sorts of tricks like letting big ticks through if they are going to hit stops and altering the spread to generate more capital at the open and close of trades. These are all the things that are hindering us in our pursuit of profits.

Good luck.