Backtester Bug

New comment
 
I was recently backtesting an EA I wrote for someone else, when I discovered it was producing odd results.
I was backtesting on H1 using 'Every tick' model, it seems to produce impossible ticks, here is one result:



And here is the M30 chart (EURUSD 2pip Spread):


One can see the entry price of 1.2781 only occurs on the 12:30-13:00 bar and the stoploss price of 1.2814 only occurs on the 12:00-12:30bar. Hence buying in at 1.2781 shouldn't get stopped out at 1.2814, yet it happens. When I run the expert it claims it is modelling the ticks on the M1 data, so it shouldn't make these ticks outside of the M30 high/lows?

I checked the tick data and found in the minute 12:30-12:31 I get 'Ask' values ranging from 1.2781-1.2828
The M1 bar for 12:30 ranges from 1.2779-1.2808

NB: using MT4 build 196
 

This problem seems more wide spread than I first thought..
To control conditions I erased all eurusd history, closed all charts except eurusd M1 and downloaded latest Alpari eurusd M1 data, then repopulated m5-h4 with period converter.

Here is an example of bad tick data you can get testing on H1 with 'Every tick' option (which should respect M1 high/low?).
The test period is 2006.07.19 to 2006.07.20, the EA is just:

Start() {
  if(TimeHour(CurTime())==16 && TimeMinute(CurTime())==1)
      Print("Sec:", TimeSeconds(CurTime()), " Bid:", Bid);
   return(0);
}


Super-imposing this on M1 chart:



This tick is inside H1 bar for 16:00.

Obviously results like this make backtester unuseful. And I have grown dependent on backtesting.. :)

 
1. Check "Visual mode" button click "Start" and see

2. Uncheck "Visual mode", check "Recalculate" button and click "Start". You'll get difference
 
I watched the visual mode and unchecked and recalulated and nothing changes.

I checked all time frames from M1-H1. Here are the maximum bid values:

2006.07.19 16:01 EURUSD,M1: MaxBid: 1.2543
2006.07.19 16:01 EURUSD,M5: MaxBid: 1.2543
2006.07.19 16:01 EURUSD,M15: MaxBid: 1.2573
2006.07.19 16:01 EURUSD,M30: MaxBid: 1.2573
2006.07.19 16:01 EURUSD,H1: MaxBid: 1.2579

It seems using 'Every tick' option that not even the nearest less timeframe is being checked. eg M15 is producing ticks that lie 30pips outside M5 data.

I tried watching the M15 visual mode all the way through (Every tick mode, recalculate checked). In the 16:00 bar it spikes up to 1.2573 before dropping to 1.2518, clearly in conflict with the M1 data (in the graph in my previous post).

PS is it just me or in visual mode is:
speed 30 = too slow using every tick
speed 31 = instant finish

Visual mode will be good when you finish the rewind button:).
 
have a look at this 9 line EA (BackTestGapChecker)

extern int   ShowGreaterThan=25;
double   ThisBid, LastBid, Gap;
int start() {
   LastBid = ThisBid;
   ThisBid = NormalizeDouble(Bid,Digits);
   Gap     = NormalizeDouble((ThisBid-LastBid)/Point,0);
   if(Gap>ShowGreaterThan) Print("Gap is=", Gap);
}  
This shows (in the Journal) where the gap between successive Bid values is excessive. It highlights your problem, and a few others (especially if you lower the "ShowGreaterThan" value).

This problem is causing havoc with my EA testing.
 
Hey, nice illustration of the problem.

There are two issues here, when testing on longer periods tick data does not respect boundaries of M1 data and also ticks gap all over the place. When I saw build 197 come through I was hoping it would provide the fix, but alas no.

If you rewrite all your EAs such that they can be tested on M1 frame it helps, ie modify all function calls that use the current period with calls that specify the exact period you wish to use.

eg If you want to test on H1,

have in header:
int tPeriod PERIOD_H1;

then replace all calls like High,Low etc with iHigh,iLow(NULL, tPeriod, ..) etc.

Then test on M1.
 
eg If you want to test on H1,

have in header:
int tPeriod PERIOD_H1;

then replace all calls like High,Low etc with iHigh,iLow(NULL, tPeriod, ..) etc.

Then test on M1.
Maybe an other way is just to recompute you H1 historical data from the M1 data with th script period_converter.
 
jlpi wrote:
eg If you want to test on H1,

have in header:
int tPeriod PERIOD_H1;

then replace all calls like High,Low etc with iHigh,iLow(NULL, tPeriod, .. ) etc.

Then test on M1.
Maybe an other way is just to recompute you H1 historical data from the M1 data with th script period_converter.

As I pointed out in my second post I already do this. Through the whole process I don't use any charts other than M1 either, least some of the brokers data gets downloaded and polute the process. The problem is not that the H1 data is inconsistant with the M1 data, I have checked that very carefully it is all in order.

I notice there are other experienced coders frequenting this forum, might I ask some share there opinion/experience with the backtester?
 
I find the lack of concreate response worrying here, what does it take to get the bugs out of the backtester, it can't be that hard!!
 
I'm doing long term backtests from 1 to 5 years. I have noticed that with both IBFX and FXDD that the backtest scans will dramatically slow down when it gets to ~ 19-6-2006 data. Whether 1 or 5 years, it will scan from beginning to ~19-6-2006, within 5 to 10 minutes. Then it will take 3-4 hours to complete backtest to present date.

Somehow, it seems that the data format changes during June '06.

Wackena
New comment