Limitation on MT5 tester?

[Daniel Whitehouse]

Hello everyone! 

I am making this EA that will essentially use correlations within pairs to make decisions. For my model to work I need to sort the data (skip the times that bars are generated on one pair and not in the other). I already had this "sorter" working so far and gave me not problems on the last couple months.  The problem begins when I use the same function to create a new model that will calculate what bars could be considered as "outliers". This required the function to go from grabbing 300-2000 bars up to 50000 bars. This process does take its time but I don't need it to run it often so it really didn't matter. I made first a script, run it, and it works exactly as expected. When I moved the script into the EA, some ifs statements were being skipped. I had this function inside "OnStart()" (it's needed to setup the EA). I moved it into the "OnTick()" and apparently that solved that issue (if it works, it works). Here is the thing that is crazy:

On both cases when running on the Strategy Tester, when it hits 12/31/2021, it stops grabbing more bars. I usually run the tests on the last couple months. I already made sure that my broker has the data and when I run the script it does grab the data before that date. It is when I run the EA on the Strategy Tester that stops at that date. Also, if I drop the EA into a chart, it works exactly like it should. Already tried running the Strategy Tester on previous months and it also stops exactly at that date. I am getting the error "Array Out of Range" . The only reason I can see is that there must be some limitation on the Strategy Tester, which I never encounter before when running EA that required larger amount of data being collected. I also thought that there may be some special scenario that I haven't accounted for but this function has cleared that scenario before. It is worth mentioning too that if the strategy tester starts close to that date (i.e 04/12/2022), it runs as expected.

Here I am going to leave the function:

   MqlRates xe[];                       //Symbol 1 Rates
   int      numberPriceDatae;      //Symbol 1 Data
   MqlRates ye[];                       //Symbol 2 Rates
   int      numberPriceData2e;          //Symbol 2 Data

   double   array_xe[];
   double   array_ye[];
   double   array_ze[];
   double   array_zxe[];

   int      secXe, secYe;
   int      iie = 0, ixe = 0,iye = 0;
   double   priceXe, priceYe;
   int      nne = Historical_Bars;
   int      xHise, yHise;


   ArraySetAsSeries(xe,true);
   ArraySetAsSeries(ye,true);
   ArraySetAsSeries(array_xe,true);
   ArraySetAsSeries(array_ye,true);

   while(iie < nne)
     {

      numberPriceDatae = CopyRates(symbol1,TF,0,ixe + 1,xe);
      numberPriceData2e = CopyRates(symbol2,TF,0,iye + 1,ye);
      
      secXe = (int) xe[ixe].time;
      secYe = (int) ye[iye].time;
      priceXe = xe[ixe].close;
      priceYe = ye[iye].close;

      Print(secXe);
      Print(secYe);
      Print(xe[ixe].time);

      if(secXe == secYe)
        {

         ArrayResize(array_xe,iie + 1);
         ArrayResize(array_ye,iie + 1);
         array_xe[iie] = priceXe;
         array_ye[iie] = priceYe;

         //if (ii >= 49000)
         // Print("Iteration = " + ii + " / PriceX = " + priceX + " / PriceY = " + priceY + " / Time = " + x[ix].time );

         ixe++;
         iye++;
         iie++;

        }

      if(secXe > secYe)
        {
         Print("lol");
         ixe++;

        }

      if(secXe < secYe)
        {
         Print("lol");
         iye++;

        }

     }

   xHise = ixe + 1;
   yHise = iye + 1;

What it does essentially is to grab x amount of bars and make sure that they are at the same time before doing any other calculations. I am getting the error before I am able to do any more calculations.

Anything that could send me to the right direction would help. Thank!

[Daniel Whitehouse]

UPDATE: I found that strategy tester is only letting me see 2 years back into the data. Not on a linear way (always a certain amount of bars) but more like, if I test on 2023, I can't access data on 2021. The same if I test on 2022, I cannot access the data on 2020. Does anybody know what may be causing this?

[Fernando Carreiro]

Your topic has been moved to the section: Expert Advisors and Automated Trading
Please consider which section is most appropriate — https://www.mql5.com/en/forum/172166/page6#comment_49114893

[Fernando Carreiro]

Daniel Whitehouse #: UPDATE: I found that strategy tester is only letting me see 2 years back into the data. Not on a linear way (always a certain amount of bars) but more like, if I test on 2023, I can't access data on 2021. The same if I test on 2022, I cannot access the data on 2020. Does anybody know what may be causing this?

At the initial date of the test period specified in the Strategy Tester settings, it only offers about 1000 bars back.

To access further back, you have to set an internal starting date in your EA, but set the test period further back in time.

[Fernando Carreiro]

PS! Also make sure that the broker actually offers that amount of historical data. Some brokers (or types of accounts) offer very little historical data.

[Alain Verleyen]

Daniel Whitehouse:

Anything that could send me to the right direction would help. Thank!

1. You are getting errors because your code is not reliable. You should ALWAYS check the returned value of CopyXXX functions, and never assume you will get what you asked for in terms of data count.

2. As already explained by Fernando, the history bars available at the start of the Strategy Tester is limited (not to 1000 bars) but to one year of data, and it depends of the timeframe used.

The testing agent downloads only the missing history, with a small margin to provide the necessary data on the history, for the calculation of the indicators at the starting time of testing. For the time-frames D1 and less, the minimum volume of the downloaded history is one year.

Thus, if we run a testing on an interval 2010.11.01-2010.12.01 (testing for an interval of one month) with a period of M15 (each bar is equal to 15 minutes), then the terminal will be requested the history for the instrument for the entire year of 2010. For the weekly time-frame, we will request a history of 100 bars, which is about two years (a year has 52 weeks). For testing on a monthly time-frame the agent will request the history of 8 years (12 months x 8 years = 96 months).

The solution was given by Fernando too.

[Fernando Carreiro]

@Alain Verleyen #: 2. As already explained by Fernando, the history bars available at the start of the Strategy Tester is limited (not to 1000 bars) but to one year of data, and it depends of the timeframe used.

I stand corrected. A bad habit from the MT4 days.

[Daniel Whitehouse]

Thanks a lot! I will try the solution that Fernando gave.

[Daniel Whitehouse]

Tested it and that solved problem. Thanks again!