Total confusion about extracting historical data

New comment
 

I am trying to generate 1-minute historical data for use in backtesting. I have been left totally confused about how MT4 handles historical data:

(1) I have tried using MQL4's "timeseries access" functions (iTime, iClose, etc). Where these functions are used on a currency pair that has not previously had a chart loaded for it (and therefore no historical data has been downloaded previously), these functions will go back from the current time by 2000-2100 data points, then just stop returning data. I am using GetLastError to check for error 4066 (waiting for new data). The failure is silent, in that no error is raised when the fuction "runs out" of data - it just starts returning zeroes, and the program carries on looping. What is the significance of this number (2000-odd)? Shouldn't these functions raise an error in this situation? Is there any way to write code that will go back in time further - a lot further - without manually opening and scrolling a chart for each currency pair?

(2) If I do manually open a chart for a currency pair that has not had a chart opened for it previously (and therefore no historical data has been downloaded previously), the time period displayed appears to correspond to the 2000-odd data points that I get via MQL4. However, I can also "force" the chart to download data further back by repeatedly hitting the Home or PageUp buttons. This does seem to have a limit, beyond which it cannot obtain further data, and this appears to be around 67,000 data points. Again, what is the significance of this number? (I have "Max bars in chart" set to 250,000.) How come the chart can go back further in time than MQL4? Is there some way to go back even further in time using a chart? (I appreciate I can download much more historical data from the History Centre, but I have read dire things about these data on this forum.)

I do feel the whole field of historical data needs better documentation by MetaQuotes.

 

Hello phoront !

Only data is available that has been downloaded from the broker or is available there.
If you want more data you simply have to wait until a longer time period has expired. My providers did not offer additional older data to me.

And you also have to do what you have already done, you have to open each chart and period periodically and scroll back in time to receive more data.
If you do not actively download historical data this way, it is not available to your programs.
But there seems to be no limit in time and amount of historical bars - my oldest data did not get lost until now, and this are tenthousands of one period of a symbol

Also the data from your provider may not always be without errors, but normaly it is no handicap for your backtests.

Hope this helps, Regards, Klaus

 

Hi Klaus, thanks for your reply. You've confirmed some of what I suspected. But it still leaves some big questions that aren't addressed by any of the documentation:

1. MQL4 seems capable of downloading ~2000 data points from the broker, but no more. This is data that has _not_ previously been downloaded from the broker. Why this number? Why so small? Why does it differ from the number of data points that can be downloaded via a chart?

2. Why am I hitting a limit of ~67,000 when downloading via a chart? Is this a limit that comes from the broker - perhaps that is all the history he makes available?

Even if I were happy to live with the higher (but still small) limit of ~67,000 data points, it is still a complete pain to manually scroll through each currency pair's chart. There has to be a better way!

Is anyone from MetaQuotes reading this thread?

\ | /
phoront Oo:
/ | \

 

Unfortunately I can't tell you about the program limitations, by myself I didn't reach such till now.

But I can give you a tip on the download of data - it can be automated done with an EA which skips back through the history of each Symbol by using commands like iHigh.
So you don't have to do it by yourself every few days.

 
phoront:
  1. MQL4 seems capable of downloading ~2000 data points from the broker, but no more. This is data that has _not_ previously been downloaded from the broker. Why this number? Why so small? Why does it differ from the number of data points that can be downloaded via a chart?
    That's what it does. If you want more, do something like this:
    double getHistory(int shift){
   int j=1;
   while(j<10 && (iClose(CurrentSymbol,ServerSyncTimeFrame,shift)==0 || GetLastError()==4066)){
      Sleep(j*1000); RefreshRates(); j++; }
}
  2. Why am I hitting a limit of ~67,000 when downloading via a chart? Is this a limit that comes from the broker - perhaps that is all the history he makes available?
    All that's provided by the broker. Keep a M1 chart open in the background. By the time your EA gets completed you'll have much more.
  3. Even if I were happy to live with the higher (but still small) limit of ~67,000 data points, it is still a complete pain to manually scroll through each currency pair's chart. There has to be a better way!
    • Open the history center to the pair you want and highlight the M1. Don't press download (that doesn't come from your broker.)
    • Scroll the listing to the oldest bar. Edit the oldest bar's time to 1999. Save and close history.
    • Right click on the chart and Refresh
    • Open the history center again and delete the oldest bar with the 1999 date. Takes 10 seconds per pair.
 

@WHRoeder: thanks for your very informative reply. A few observations:

- Your function needed a return() statement.

- It doesn't successfully get more than ~2000 data points, so the RefreshRates() call doesn't seem to add any benefit. (I'm trying to code a script as opposed to an expert advisor.)

- Your method of forcing the download of greater than ~2000 data points using the History Centre is cool! Still wishing I could achieve this programmatically...

phoront

 
phoront:

@WHRoeder: thanks for your very informative reply. A few observations:

- Your function needed a return() statement.

- It doesn't successfully get more than ~2000 data points, so the RefreshRates() call doesn't seem to add any benefit. (I'm trying to code a script as opposed to an expert advisor.)

- Your method of forcing the download of greater than ~2000 data points using the History Centre is cool! Still wishing I could achieve this programmatically...

phoront


Did you ever come up with a solution this?  I'm trying to do the same thing myself. 
 
ambtn723:

Did you ever come up with a solution this?  I'm trying to do the same thing myself. 
You can't get more data than your Broker is willing to give you . . .  if you need more go elsewhere.
New comment