problem with ibarshift getting values from higher timeframes in strategy tester
Please read the posts at the following related thread: https://www.mql5.com/en/forum/160159
Although not exactly the same question, the reasons are the same. Pay special attention to the posts by the user "jjc".
hi fmic,
as always you understand the problem and you found the right words / link ;) i did a search before and i found a lot of problems with iBarShift in backtester, but i didn't found this link ;). thanks! there are some details that i don't understand:
1) i have a full history, all higher timeframes have been created from m1-data, that means i got all the bars in every timeframe M1, ... , H1, H4, D1 and so on...? i think the problem is not that there are data / files missing....?
2) iBarShift works like a charm, in the moment you add the indicator to the running/stopped chart it is working like it should, until a new bar is created in the higher timeframe. this is the moment when iBarShift fails, before that - everything is fine? and - if i recompile the indicator (when the visual backtest is running) iBarShift works again, until a new bar in the higher timeframe appears?
recompile, it works - new h4 bar - it fails
recompile, it works - new h4 bar - it fails
and so on...
the indicator that shows me the problem is a simple high/low indicator, it shows highs/lows from higher timeframes, because of that i can't test it...? i can calculate a new period length to get the same results, but that is not what i need at the moment. also the indicator draws a correct history, i add it to the chart, all history data of the indicator highs/lows are correct = iBarShift works, i get no error.
i'm still not sure how to solve the problem, why are there moments when it is working and a few ticks later it fails? it sounds like metaquotes forget to shift the buffer in memory for other / higher timeframes?
in another project i'm facing the same problem. imagine in a channel, backtest on H1 you have a candle with a high[1] above the upper band and low[1] below the lower band. i don't know what was first, the upper breakout or the lower, so i'm going down to M1 and i can calculate what was first. i can't backtest the hole system because of that iBarShift Problem!? on forward tests there is no problem, but you need months, years to test it...
any workaround? what do you think?
regards ;)
another example, my channel indicator is attached to chart while the dummy-ea is running in H1 visual-backtest.
if a candle hits both, upper and lower band - it is using M1 data to calculate the correct direction...
the backtest was running, the big candle was @ shift 0 (it is now at 1) and i get the error iBarshift = -1.
if i recompile it now, there is no error for iBarShift, why it works after recompiling?
>>>>>>>>>>>>>>>> normally here has to be a new error after the recompile ? 2013.07.03 11:01 Custom indicator channel_v20 GBPUSD,H1: loaded successfully 2013.07.03 11:01 Custom indicator channel_v20 GBPUSD,H1: removed 2013.07.03 11:00 channel_v20 GBPUSD,H1: M1ChannelBreak() | ERROR | i 1 | ibs -1 2013.07.03 10:59 channel_v20 GBPUSD,H1: M1ChannelBreak() | ERROR | i 0 | ibs -1 2013.07.03 10:59 channel_v20 GBPUSD,H1: M1ChannelBreak() | ERROR | i 0 | ibs -1 2013.07.03 10:59 channel_v20 GBPUSD,H1: M1ChannelBreak() | ERROR | i 0 | ibs -1 2013.07.03 10:59 channel_v20 GBPUSD,H1: M1ChannelBreak() | ERROR | i 0 | ibs -1
why there is no error anymore, the backtest is still running! that means that iBarShift is working after recompile!?
- i have a full history, all higher timeframes have been created from m1-data, that means i got all the bars in every timeframe M1, ... , H1, H4, D1 and so on...? i think the problem is not that there are data / files missing....?
- iBarShift works like a charm, in the moment you add the indicator to the running/stopped chart it is working like it should, until a new bar is created in the higher timeframe. this is the moment when iBarShift fails, before that - everything is fine? and - if i recompile the indicator (when the visual backtest is running) iBarShift works again, until a new bar in the higher timeframe appears?
... i'm still not sure how to solve the problem, why are there moments when it is working and a few ticks later it fails? it sounds like metaquotes forget to shift the buffer in memory for other / higher timeframes? ... any workaround? what do you think?
It seems, that you did not read or did not understand the posts that "jjc" wrote at the the thread mention. So, I will quote him here, so that you take note of the most import part of it:
jjc:
Are you doing a backtest of the indicator, or are you doing a backtest of an EA and then manually adding the multi-timeframe indicator onto the backtesting chart for the EA?
The latter doesn't work properly in MT4. You can add indicators to the backtesting chart of an EA, but use of timeseries functions such as iBarShift and iClose within such a separate indicator breaks out of the backtesting sandbox.
For example, do a visual-mode backtest of any EA on a period such as H1, and then drop the following indicator onto the backtesting chart. It should display the latest simulated close price within the backtesting environment, just like it shows the live price on a live chart. But it doesn't.
Doing a backtest of this indicator, rather than doing a backtest of an EA and adding the indicator to the EA's chart, does work correctly.
another example:
2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 50 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 97 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 194 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 242 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 264 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 759 | ibs -1 2013.07.05 12:00 signal_v11 GBPUSD,H1: GetLastChannelBreak() | ERROR | i 885 | ibs -1
picture 1 shows the backtest in H1, at the beginning i added my indicator to find the highs / lows. if there is a oversize candle that breaks both, highs and lows i go down to M1 and search for first / last break to determine the underlying direction.
to determine the correct m1 start and end points i'm using iBarShift()...
as you can see, the pink lines are at the points where iBarShift fails with -1... the terminal shows some errors, the bar shift of the error and ibs result = -1.
the backtest is paused - to get another picture after i hit "compile" for the currently loaded indicator in metaeditor.
now i hit "compile", my indicator recalculates the hole history from MaxBars (i.e. 3000) down to bar 0 and i get this
>>>>>>>>>>>>>>>> after compiling are no errors in history , iBarShift works now!
2013.07.05 12:32 Custom indicator signal_v11 GBPUSD,H1: loaded successfully
2013.07.05 12:32 Custom indicator signal_v11 GBPUSD,H1: removed
in picture 2 you can clearly see that there is no error anymore, iBarShift did it's job, the indicator was able to get the underlying M1 data for the oversize H1 candles and calculates the direction correctly.
also there is no error in the terminal anymore, so the question is - why does it work after adding / reloading or recompiling the indicator?
again it seems to be a bug, the more i see it i come to the conclusion that they do not shift the internal arrays for other timeframes of added indicators during runtime.
if you add / reload / or compile (reload) the indicator, they refresh the internal arrays and everthing is fine.
maybe because of performance issues, maybe there is an error or they simply forget it... in case 2 + 3 a bug report to metaquotes should be created, for me it simple does look like iBarShift is bugged?
what does that mean? lets say the indicator "breaks" out of the backtesting sandbox, to where ? in case it is getting wrong values, where from? in case it is getting no values, why does it work when i add / reload / recompile the indi ?
The reason you don't understand is because you don't know the basics of programming. You should first learn about general coding principles such as what a "sandbox" is.
I will try to make it more simple so you can understand:
- It does not work that way!
- It is not a bug!
- It is simply the limitations of the testing environment (i.e. the "sandbox")!
That is why, there is a testing "sandbox" for Indicators and another one for Expert Advisors. You should not be testing Indicators, using the Expert Advisor "sandbox".
Test your Indicator in the "Indicator" test mode only!
i just can't do a visual backtest of a trading system (containing a lot of more stuff) -> by viewing 1 indicator...
if i'm testing an indicator -> there are no trade operations possible and i can only test 1 indi at time
if i'm testing an expert -> there are no indicators or they are showing wrong values (if they use values from other timeframes)
sorry but you know nothing about me, my background, my work, projects i have done and so on... this remark was unnecessary... there is no reason to be angry or attack me, i just ask simple questions?
i simply can't do a visual backtest of a trading system (containing a lot of more stuff) -> by viewing 1 simply indicator...
I was not being angry at all! I was not attacking you either. I was simply stating the obvious because you said and I quote:
... what does that mean? lets say the indicator "breaks" out of the backtesting sandbox, to where?
This, your statement, simply indicated to me that you lacked the necessary knowledge and experience, and that your programmings skills are that of a beginner. There was no malice or aggression in my statement in any way.
I have tried repeatedly to show you that what you are trying to do is beyond the limitations of the system, but you keep asking the same questions trying to justify that there must be a bug and that it needs to be fixed or a workaround proposed.
We are just users of MetaTrader, so if you feel that it is not working as it should, then please address the developers and not us users.
If you cannot back-test your trading system with the limitations of the current implementation of MetaTrader's Strategy Tester then you will have to develop your own mechanisms!
where is the answer to the question im asking again and again?
to explain it better... imagine there is a so called "sandbox" when i'm testing an expert advisor... the ea in backtest runs in an isolated area with complete own data... i understand that, the backtester creates a testing environment that is separated from the normal quotes, files and so on... the ea runs in its own thread... the biggest problem with threads is sharing data between them, you need to synchronize the access, do a lot of stuff, lock the memory maybe with a mutex to prevent access violations...
in case the ea runs in a own separated sandbox there would be normally NO WAY - NEVER EVER - to get corresponding values with the attached indi... right? if it runs in a sandbox, and there is no way to get any data from inside this isolated area, why does iBarShift work and can get correct values after reloading the indi or compiling it? or is the indi in the sandbox and sometimes not ? i think it is a problem of the threads the ea is running in and the additional indicators (in own threads) you add to chart... maybe they do not synchronize indi threads at runtime to get better performance....
but if you need mtf indicators or calculations - there still is hope and a way... you can test all indicators one by one, if they run correctly you need to load them inside the ea with icustom and now you're able to test the hole system... but currently it seems that there is no way to do a visual backtest of an ea with ATTACHED mtf indicators...
I did answer your questions, but you keep INSISTING and REFUSE to accept it. So, I leave you to your own devices!
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
hi guys,
first i need to say that i use a separate folder for back-testing, in my "mt4/history/" is a folder "backtest", the full path is "Program Files (x86)/MetaTrader/history/backtest". if i want to use mt4 with broker data, i login to normal account / demo-account, if so the history is taking from the brokers folder for example "Program Files (x86)/MetaTrader/history/Broker-Demo01".
if i want to backtest something i login to the "backtest" server, in the login form you can enter "backtest" as a server name (any user / pwd) and the mt4 takes the history from that server/folder. now you're not connected to any broker, it is like a offline mt4. but! you can download history from metaquotes, no problem.
if i'm on the "backtest" server i can go to the history-center and download for example USDJPY M1, all higher USDJPYxxxx.hst files (from M1 to MN1) are created in "Program Files (x86)/MetaTrader/history/backtest".
in this way you never merge broker data and metaquotes data, i got 90% modeling quality with 0 errors and my metaquotes data are separated from broker data. i've explained that because damaged fxt or hst files can create errors, i think that is not my problem...
the problem is the following, if i use iBarShift to get the bar shift for any other then bar 0 of higher timeframes (in an indicator attached to the running visual backtest) it works - for the bar it was loaded on. if there is a new bar in the higher timeframe all calls to bars > bar 0 fail with result -1. i know that iBarShift has a parameter called "exact", i set this to TRUE to see if it really finds the correct bars. with a clean history - all higher timeframes are build from m1 data - there normally can be no error!
attached you'll find a simple indicator and an empty dummy-ea. you can test it really simple, load the ea, start a visual backtest on H1. if it is running add the indicator to the chart, maybe slow down the speed so you can see the logs in the terminal.
the indicator is only reading the iBarShift value of the PERIOD_H4 timeframe, remember, you're on the 1 hour chart!
i search the h4 bar shift by the Time[1] of the h1 chart.
for the first h4 bar it is working, but if a new h4 bar is created, iBarShift fails.
is it a bug, a feature, what is wrong ?
for me it looks like the higher timeframes are not modeled correctly, bar 0 works, all other bars are not carried forward or "do not exist"?
you can test it in every tick mode or control points for faster tests!
i also know that there is a option to test indicators, but that is not my question ;)