Downloading minute bars

[lori]

I am trying to download historical data in order to back-test some strategies. For some reason the MT5 Strategy Tester was not able to get the minute bars which I was requesting, so I started looking into how to force MetaTrader to download the historical data.

I found an MT5 script online here https://www.mql5.com/en/docs/series/timeseries_access; according to this article accessing the bars using CopyTime/CopyRates I should be able to forcibly obtain the data.

// copying of next part forces data loading
int copiedTimes = CopyTime(symbol, period, count, arraySize, times);
int copiedRates = CopyRates(symbol, period, count, arraySize, rates);

I wrote a program which does just this - it downloads every minute bar for the period 2000 - today, and writes the date and OLHC values to a file (source code below)

I was surprised to find that 2000 - 2008 only has daily data; 2009 has hourly, and 2010 has hourly up until 7th April, and then minute bars from then until today.

So I then contacted my broker (Alpari UK) and their response was: 

No unfortunately this is not possible, it’s a colossal amount of data you are trying to download. On MT5 it is possible on some currency pairs, but even then the data can sometimes be incomplete if you are trying to download such a vast amount of information which we do not support. 

They suggested I try contact MetaQuotes. 

I just find it so hard to believe that this is not supported.

If I go to the history folder (MetaQuotes\Terminal\...\bases\AlpariUK-MT5\history\EURUSD), the entire history for 2011 is 15MB. For ~15MB of data per year, I don't think it's that colossal to be honest. 

In any event, I'm finding it hard to believe that Alpari and/or MetaTrader doesn't support back-testing before 2010, because it's just such a fundamental part of automated trading.

Is this a technical shortcoming, or is it the way Alpari has configured their server? 

On a related note, does anyone know of an MT5 broker who does support minute bars back to 2000? 

My script: 

#property script_show_inputs

input string          InpLoadedSymbol="GBPUSD";   // Symbol to be load
input ENUM_TIMEFRAMES InpLoadedPeriod=PERIOD_M1;  // Period to be load
input datetime        InpStartDate=D'2000.01.01'; // Start date

#include <Files\FileTxt.mqh>

void OnStart()
{
    Print("Start load",InpLoadedSymbol+","+GetPeriodName(InpLoadedPeriod),"from",InpStartDate);

    int res=CheckLoadHistory(InpLoadedSymbol,InpLoadedPeriod,InpStartDate);
    switch(res)
    {
        case -1 : Print("Unknown symbol ",InpLoadedSymbol); break;
        case -2 : Print("Requested bars more than max bars in chart"); break;
        case -3 : Print("Program was stopped"); break;
        case -4 : Print("Indicator shouldn't load its own data"); break;
        case -5 : Print("Load failed"); break;
        case  0 : Print("Loaded OK"); break;
        case  1 : Print("Loaded previously"); break;
        case  2 : Print("Loaded previously and built"); break;
        default : Print("Unknown result");
    }

    datetime firstDate;
    SeriesInfoInteger(InpLoadedSymbol,InpLoadedPeriod,SERIES_FIRSTDATE,firstDate);
    int bars=Bars(InpLoadedSymbol,InpLoadedPeriod);
    Print("First date ",firstDate," - ",bars," bars");
}
//------------------------------------------------------------------

int CheckLoadHistory(string symbol,ENUM_TIMEFRAMES period,datetime startDate)
{
    if (symbol==NULL || symbol=="") symbol=Symbol();
    if (period==PERIOD_CURRENT)     period=Period();

    // check if symbol is selected in the MarketWatch
    if (!SymbolInfoInteger(symbol, SYMBOL_SELECT))
    {
        if (GetLastError() == ERR_MARKET_UNKNOWN_SYMBOL) 
            return -1;
        SymbolSelect(symbol,true);
    }
    
    datetime firstDate=0;

    CFileTxt file;
    file.Open("out.txt", FILE_WRITE);

    #define arraySize 100

    datetime times[arraySize];
    MqlRates rates[arraySize];
    
    // max bars in chart from terminal options
    int maxBars = TerminalInfoInteger(TERMINAL_MAXBARS);
    
    // load symbol history info
    datetime firstServerDate = 0;
    while (!SeriesInfoInteger(symbol, PERIOD_M1, SERIES_SERVER_FIRSTDATE, firstServerDate) && !IsStopped())
        Sleep(5);
        
    // fix start date for loading
    if (firstServerDate > startDate) 
        startDate = firstServerDate;
        
    if (firstDate > 0 && firstDate < firstServerDate)
        Print("Warning: first server date ",firstServerDate," for ",symbol," does not match to first series date ",firstDate);
        
    if (file.WriteString("date,open,low,high,close\n") <= 0)
        printf("error writing to file");

    // load data step by step
    int failCount = 0;
    int count = 0;

    while(!IsStopped())
    {
        // wait for timeseries build
        while(!SeriesInfoInteger(symbol, period, SERIES_SYNCHRONIZED) && !IsStopped())
            Sleep(5);
            
        // copying of next part forces data loading
        int copiedTimes = CopyTime(symbol, period, count, arraySize, times);
        int copiedRates = CopyRates(symbol, period, count, arraySize, rates);

        // write times and rates to file
        for (int i = copiedRates - 1; i >= 0; --i)
        {
            string line = TimeToString(times[i]) + "," + DoubleToString(rates[i].open) + "," + DoubleToString(rates[i].low) + "," + DoubleToString(rates[i].high) + "," + DoubleToString(rates[i].close) + "\n";
            if (file.WriteString(line) <= 0)
                printf("error writing to file");
        }
        
        if (copiedRates > 0)
        {
            // check for data
            if (times[0] <= startDate)  
                return 0;
            if (0 + copiedTimes >= maxBars) 
                return -2;
                
            failCount = 0;
            count += arraySize;
        }
        else
        {
            // no more than arraySize failed attempts
            failCount++;
            if (failCount >= arraySize) 
                return -5;
            Sleep(10);
        }
    }
    return -3;
}
//------------------------------------------------------------------

string GetPeriodName(ENUM_TIMEFRAMES period)
{
    if  (period==PERIOD_CURRENT) 
        period=Period();
    switch(period)
    {
        case PERIOD_M1:  return "M1";
        case PERIOD_M2:  return "M2";
        case PERIOD_M3:  return "M3";
        case PERIOD_M4:  return "M4";
        case PERIOD_M5:  return "M5";
        case PERIOD_M6:  return "M6";
        case PERIOD_M10: return "M10";
        case PERIOD_M12: return "M12";
        case PERIOD_M15: return "M15";
        case PERIOD_M20: return "M20";
        case PERIOD_M30: return "M30";
        case PERIOD_H1:  return "H1";
        case PERIOD_H2:  return "H2";
        case PERIOD_H3:  return "H3";
        case PERIOD_H4:  return "H4";
        case PERIOD_H6:  return "H6";
        case PERIOD_H8:  return "H8";
        case PERIOD_H12: return "H12";
        case PERIOD_D1:  return "Daily";
        case PERIOD_W1:  return "Weekly";
        case PERIOD_MN1: return "Monthly";
    }

    return "unknown period";
}
//------------------------------------------------------------------

 

[phi nuts]

Please read this https://www.mql5.com/en/forum/7028#comment_216914 and read the whole topic.

[lori]

Thanks phi.nuts - I have read that topic - and it would seem that guy has the exact same problem...

So because my broker decides not to have historical data before a certain date, then there is no way for me to get that data, and backtesting becomes impossible.

This is very bad news, and it looks like I will have to look for another trading platform where this problem does not exist.

[phi nuts]

lori:

Thanks phi.nuts - I have read that topic - and it would seem that guy has the exact same problem...

So because my broker decides not to have historical data before a certain date, then there is no way for me to get that data, and backtesting becomes impossible.

This is very bad news, and it looks like I will have to look for another trading platform where this problem does not exist.

Just change the broker, add MetaQuotes to your server list, and it will download more historical data, just like what I've read from the link there.

[lori]

Yes, I guess that's somewhat of a solution, but it doesn't seem very robust wouldn't you say?

The points that MaxTrader makes here are still valid no? 

[NyemaSanya]

lori:

Yes, I guess that's somewhat of a solution, but it doesn't seem very robust wouldn't you say?

The points that MaxTrader makes here are still valid no? 

Hi Lori,

I am on Alpari and it has data