Strategy Tester slowing down during test

Ian Venner 2011.10.28 14:57

I know this topic has been taked about before but I didnt find a difinitive answer, I have been running some 1yr and 18month backtests the first few months are executed speedily but the test slows down considerably after that until at the end of an 18 month test each month is taking several minutes

Here is some excerpts from my tester log over a 9 month period

Tthe first 3 month period tester executes 67 trades takes a total of 26 seconds

The second 3 month period tester executes 144 trades takes a total of 192 seconds I cant imagine what could be different between that first three month period and the second, obviously the EA processed hundreds of thousands of ticks in that first three month period and appeared to perform correctly so what could be so different after that ?

The third 3 months period tester executes 147 trades takes a total of 341 seconds now even slower than that second three month period could this be due to a memory leak slowing down my PC ?

The tester log doesnt show anything unusual, there are usually two entries per trade, except on occasion when a trade was closed due to TP or SL and the EA tried to close it before discovering it is already been closed. I havent figured out what is the way to avoid that, other than scanning the Orders pool on every tick which I'm sure really would slow down the tester considerably more.

11:02:41 2010.06.16 01:00 MACD60_IDF3 v1-00 EURUSD,M15: open #1 sell 0.05 EURUSD at 1.2321 sl: 1.2374 tp: 1.2274 ok

11:02:45 2010.07.14 06:00 MACD60_IDF3 v1-00 EURUSD,M15: open #20 sell 0.05 EURUSD at 1.2724 sl: 1.2777 tp: 1.2677 ok 4seconds

11:02:58 2010.08.13 04:30 MACD60_IDF3 v1-00 EURUSD,M15: open #54 buy 0.20 EURUSD at 1.2838 sl: 1.2785 tp: 1.2885 ok 13 seconds

11:03:07 2010.09.15 01:00 MACD60_IDF3 v1-00 EURUSD,M15: open #67 sell 0.80 EURUSD at 1.2988 sl: 1.3041 tp: 1.2941 ok 9 seconds

11:03:15 2010.10.15 02:45 MACD60_IDF3 v1-00 EURUSD,M15: open #77 sell 0.05 EURUSD at 1.4029 sl: 1.4082 tp: 1.3982 ok 8 seconds

11:04:33 2010.11.15 02:15 MACD60_IDF3 v1-00 EURUSD,M15: open #144 sell 0.10 EURUSD at 1.3718 sl: 1.3771 tp: 1.3671 ok 78 seconds

11:06:19 2010.12.15 02:15 MACD60_IDF3 v1-00 EURUSD,M15: open #211 sell 0.06 EURUSD at 1.3352 sl: 1.3405 tp: 1.3305 ok 106 seconds

11:08:02 2011.01.14 01:00 MACD60_IDF3 v1-00 EURUSD,M15: open #266 sell 3.84 EURUSD at 1.3349 sl: 1.3402 tp: 1.3302 ok 103 seconds

11:10:36 2011.02.15 03:30 MACD60_IDF3 v1-00 EURUSD,M15: open #322 buy 0.04 EURUSD at 1.3516 sl: 1.3463 tp: 1.3563 ok 154 seconds

11:12:00 2011.03.15 06:15 MACD60_IDF3 v1-00 EURUSD,M15: open #358 sell 0.05 EURUSD at 1.3912 sl: 1.3965 tp: 1.3865 ok 84 seconds

William Roeder 2011.10.28 16:46 #1

SDC:
I know this topic has been taked about before but I didnt find a difinitive answer, I have been running some 1yr and 18month backtests the first few months are executed speedily but the test slows down considerably after that until at the end of an 18 month test each month is taking several minutes

You have one or more indicators that are recalculating all bars, or you are creating many objects.

Ubzen 2011.10.28 16:53 #2

My room computer is older but this is where I do allot of my work. With that said, few things I've noticed which slows down my testing on mql4.

***Calling iCustom Indicators: I've Never Written any Indicator since I've been on here. What I ended up doing is taking my two favorite indicators, lifting the codes, and placing them directly into the EA.

***Looping through Orders: I make every attempt to break out of the Loop when I've got what I'm looking for. Then I make sure to Save the state of the Order-Pool, notably OrderTotal_With-X-Magic or Time-Of-Last-Check.

***Yeah, Objects: I usually draw them the first time I changes parameters of the object. When I need a speedy test I use commands like if not visual mode ... dont draw objects.

***Once-per-Bar: Be very picky about what Information you run every Tick. The less Information you run per tick the better. Don't run indicator codes within EA every tick for example. I find once-per-M5-bars to be sufficient for most of my systems.

Migration - Virtual Hosting How to Purchase an Additional Features - Price

Ian Venner 2011.10.28 17:53 #3

the EA on test is using one indicator I dont believe it is recalculating all the bars. There are no objects created other than the usual arrows drawn by the tester at the open and close of a trade. As there are only a few hundred trades I wouldnt think that could be the problem.

Whatever the EA and indicator combination does, by nature it has to do the the same thing repeatedly on every tick and every trade signal, so theoretically if there is a problem with the code logic, surely it would be slow from the beginning and remain the same ?

I dont see why it should get progressively slower ?

I should add at one point closer to the end of an 18 month backtest both of my CPU cores were running at 100% and stuck there for some time while the tester was very slow. It makes me think the EA is doing more of something as the test progresses, I'm going to run it again this time with no tester arrows, if it isnt that could the tester log file be the problem ?

Here is the main code of the indicator I dont think it should be recalculating all the bars.

int start()
  {
   int i,j,k,limit,mins;
   int ctf=Period();
   int counted_bars=IndicatorCounted();
   limit=Bars-counted_bars-1;
   for(i=limit; i>=0; i--)
    {mins=StrToInteger(StringSubstr(TimeToStr(Time[i],TIME_MINUTES),3,0));
     int iH1 = iBarShift(NULL,TimeFrame,Time[i]);
     for(j=0,k=0; j<=mins; j+=ctf, k++)
      {MacdBuffer[i+k]=iMA(NULL,TimeFrame,Fast,0,MODE_EMA,price,iH1)
                      -iMA(NULL,TimeFrame,Slow,0,MODE_EMA,price,iH1);
    } }
   for(i=0; i<limit; i++)
    SignalBuffer[i]=iMAOnArray(MacdBuffer,Bars,Signal,0,MODE_SMA,i);
//---- done
   return(0);
  }

Fractals - Bill Williams' Parabolic SAR - Trend MetaTrader 5 Built-in Trading

William Roeder 2011.10.28 19:14 #4

mins=StrToInteger(StringSubstr(TimeToStr(Time[i],TIME_MINUTES),3,0));
     int iH1 = iBarShift(NULL,TimeFrame,Time[i]);
     for(j=0,k=0; j<=mins; j+=ctf, k++)
      {MacdBuffer[i+k]=iMA(NULL,TimeFrame,Fast,0,MODE_EMA,price,iH1)
                      -iMA(NULL,TimeFrame,Slow,0,MODE_EMA,price,iH1);
    }

extremely inefficient.

int      iH1    = iBarShift(NULL,TimeFrame,Time[i]);
double   macd   = iMA(NULL,TimeFrame,Fast,0,MODE_EMA,price,iH1)
                - iMA(NULL,TimeFrame,Slow,0,MODE_EMA,price,iH1);
int      iChart = iBarShift(NULL, 0, iTime(NULL,TimeFrame,iH1));
for(; iChart >= i; iChart--) MacdBuffer[iChart] = macd;

Ian Venner 2011.10.28 21:24 #5

I did it that way so it would work the same way on live prices as it does on history but even though your way is faster code, i still dont see how that would explain the tester getting progressively slower during the test. My slower code should make it uniformly slower throughout the test, not fast at the beginning and then gradually slower

Optimization Types - Algorithmic Code profiling - Developing Parabolic SAR - Trend

Ian Venner 2011.10.28 22:16 #6

OK I figured this out the one thing that takes progressively longer to do as the test progresses is scanning through the history pool, my EA accesses the history pool to get the lot size of the last order, and because I read, the history pool is not guarenteed to be in sequential order I wrote code to check every order in the pool to see which is the last order, by the end of the test that means checking through the list of several hundred closed orders.

So, I am either going to write code to get the profit and lot size of orders in my close orders function just before closing them, or run some tests to see if the highest numbered order in the history pool of the correct type is ever anything other than the last closed order. The former should work for orders closed by the EA, but it would also mean keeping up with the orders that are about to be closed because they have met their SL or TP levels.

Real and Generated Ticks History - MetaTrader 5 Depth of Market -

Ubzen 2011.10.29 00:51 #7

Oh, lets add one more to the list. I rarely use History-Pool Searches now-a-days. For getting value of last-order which is fairly common for allot of systems, I Save the value of LastOrderSize in Static/Global Variable. If I have something very Important and don't wanna ever search the history for it, I place it in a Terminal Global Variable.

I mean I understand that there may be times when there's valid reasons to search history, and that this may not be the best approach for every logic. But for the sake of testing which could be repetitive and time-consuming, I'd put the least amount of bullet proofing codes in. Example: I'd assume that ordertotal-1 is the last order.

Looping is a great method for building an EA which can resume where it left off during a restart...problem is excessive looping & function-calls will slow tester down. Terminal Variables help reduce the need to Loop for certain types of information during a restart. Looping helps reduce the need for excessive Terminal Variables.

I'm still in the process within my head trying to define When and Where to use What.

How to Purchase an MetaTrader 5 Built-in Trading Optimization Types - Algorithmic

Ian Venner 2011.10.29 03:58 #8

I'm going to avoid looping through the history too, but when it is neccesary why do we need to do OrdersHistoryTotal() - 1 ?

edit ok i dont know why i asked that i must be getting tired lol obviously the first order is held in the zero index of the order pool array ...

Managing Pending Orders - Managing Pending Orders - Extended Authentication - Accounts

Ubzen 2011.10.29 04:11 #9

SDC:
I'm going to avoid looping through the history too, but when it is neccesary why do we need to do OrdersHistoryTotal() - 1 ?

We don't need to. That was just an example. I always loop down. Tho I haven't done history searches for a while I believe OrdersHistoryTotal() - 1 will select last orders first (in the tester.. should work live also unless your history-pool is being filtered). That way I just -break- after I get last order info. If you're looking for something like the size of all orders in history then you'll have to loop through all trades.

Managing Pending Orders - Managing Pending Orders - Opening and Closing Positions

blackmore 2011.10.29 04:46 #10

SDC: