Can a buffer update in the middle of two commands? - page 4

New comment
 
Fernando Carreiro #:

Thank you 🙏 ... You are welcome ... I wish you good coding luck 👍.

The same to you!

I'll conclude the thread with this:

I've corrected a bug in my previous code, now, I see that 99.9% of the time, the EA is quick enough to collect data from multiple buffers for multiple timeframes on each of the traded symbols without a new bar forming while computations are taking place. But it's not 100% of the time. So I can assure everyone that checking the result of iBarShift with exact=true before and after collecting data is essential if you want to collect data from previous closes before being sure a new bar opens. This fixes the issue mentioned in the post title as it allows you to restart the data collection on the odd 0.1% chance that a new tick screws up the synchronisation by forming a new bar in the middle of two buffer query commands (which results in shifting your buffer's value by 1 index).

I also have to dissagree, after all new tests, that this adds complexity to the EA. In truth, I think it simplified things as you go from being forced to control for a bar open on all your traded symbols to only having to control for a bar open on the symbol sending ticks to OnTick which simplified your OnTick function immensely. Also, it means there is less lag in your EA since you can act before all new bars are formed, which is even more useful if some symbols are slow to deliver ticks. And while you do this, you can keep working with indicators.

This makes your multi-symbol EA more simple than if you were going to control for a bar open on all symbols and wait for each symbol to have their new opens before making programmatic decisions. So when compared with that method, all you need if you simplify the process like I do is 2-3 lines of code in your buffer copy function to check for a valid synchronization.

Fernando's method has some merit but I can't confirm that it's 100% fail proof since I haven't tested it yet. Thanks to everyone for your ideas as I was trying to work this out! And good coding to all!

 

Fernando Carreiro #:

On every new bar detection/pseudo-event (OnTick/OnTimer), I collect the bar data first (current and previous bar only, per symbol), then decide if the current bar is new and the previous one closed, or if it the current bar is stale and its duration lapsed, or even if the duration of several bars has lapsed with no updates. And based on this, I carry out the incremental calculations.

I hope you don't mind me saying, but I think you're still exposing yourself to failed synchronizations with this approach if you don't add a validation command like the one I worked on in this post. You see, I let my experimental EA run for 12 hours and it copies 2 buffers from 28 symbols and 19 timeframes every minute. And everytime it does, it validates if anything changed in between two commands that need to be synchronized to be valid and if it detects a change, it re-processes the original commands. The result of this are surprising, after 12 hours, the EA only detected 7 instances of failed synchronization. The point is, if you're dealing with current bars that are stale with a lapsed duration, in other words, bars that are milliseconds away from switching from current to previous as soon as a tick comes in, you will eventually get a synchronization error. But this will happen so rarely that unless you test and keep records of it, you're probably never going to notice.

Even if all you do is:

1. Receive an OnTimer event

2. CopyRates(symbol,timeframe,0,2,ringbuffer)

3. if (iBarShift(symbol,timeframe,TimeCurrent(),true) == -1) { current_bar_duration_is_lapsed = true; }

The reality is that if the timing is perfect, the current bar copied on step 2 could be lapsed, but when you check if it's lapsed on step 3, the EA could tell you it's not lapsed. That will happen if a new tick is received between step 2 and 3. So your EA will assume ringbuffer[1] is the previous close, when in fact ringbuffer[0] is the true previous close.

I realize the chances of this happening might be 1 in 100000, but unless there's something I don't understand about your approach, I'm pretty sure it's not zero.

But then you could fix that by doing this:

void on_timer_event_detected() {
        for (int i = 0; i < 1; i++) { 
                int iBarShiftResult = iBarShift(symbol,timeframe,TimeCurrent(),true);
                CopyRates(symbol,timeframe,0,2,ringbuffer);
                if (iBarShiftResult  == -1) {
                        current_bar_duration_is_lapsed = true;
                } else {
                        current_bar_duration_is_lapsed = false;          
                }
                if (iBarShiftResult != iBarShift(symbol,timeframe,TimeCurrent(),true)) { 
                        i--; 
                }
        }
        if (current_bar_duration_is_lapsed) {
                Print("ringbuffer[0]'s bar duration is lapsed");
        } else {
                Print("ringbuffer[0]'s bar duration is not lapsed");
        }
}                
Maybe I'm wrong because I didn't see your code and you're doing some things that I haven't tried yet. But it seems to me that the odds of failed synchronizations happening are so small that it's easy to presume it never happens even though all the experiments I did this week showed me that the odds are not 0% (even with your approach), and that it's actually easy to add 2-3 lines of code to prevent it. So maybe this changes almost nothing in the long run and we don't notice the difference, but I would still prefer being 100% sure. Just my 2 cents. If I'm wrong, no need to convince me, I'll trust you! I didn't see your code so I can only guess... I know you are coding more advanced EA's than what I'm used to ;)
 
Jeepack #:I hope you don't mind me saying, but I think you're still exposing yourself to failed synchronizations with this approach ...

Not at all! I don't have that issue, and I quote from post #28: "... As long as I keep a state machine for each symbol, and do the incremental calculations, my data for each and every symbol will always be correctly aligned and synchronised."

I also don't need the use of iBarShift ever. As I collect each symbol's data, I know the time of those bars from the collected rates, and I can easily "fill in" missing bars to align them. There are often missing bars or gaps in time, especially on the lower time-frames. I make sure that the whole array/collection of symbol data is always aligned and that the data is added and calculations done in an incremental way.

In fact, normal indicators, don't account for missing bars, and do the calculations based on the next available bar, even if there might be 1, 2, or what ever number of bars missing in between. This can make aligning Indicators for multi-symbol quite a challenge. The way I do it, all the Indicator calculations are aligned and take into account missing bars.

I also, take into account the session trading times so as not to "fill in" bars that are not supposed to be filled in (e.g. weekends). Fortunately I only have EAs where the symbols used all share the same trade session times, but if I had such a case of different trade sessions, I would probably do things based on the lowest common denominator in regards to the trade sessions (but I guess that would depend on the strategy).

Also, I usually try to restrict myself to only using EMAs when ever possible, so that I don't need ring buffers at all, but when I need to do keep track of other things like SMAs or Donchian channels, then my ring buffers are only as large as the the period or depth required for those types of indicator calculations.

I hope that clarifies my method for you.

 
Fernando Carreiro #:

As I collect each symbol's data, I know the time of those bars from the collected rates, ...

This is the only part that I keep struggling with. There's probably something really basic I'm not thinking of.

****EDIT: oh wait, I just figured it out, I just don't usually use MqlRates so I hadn't realized that CopyRates isn't the same as CopyBuffer, now things are starting to make sense. It's as if you were querying a multi-dimensional array in a way. After you've copied it, you can then access things like the Open, Close, and bar open time from the data you previously copied. I didn't realize that, this is why I didn't understand how you had no synchronization issue.

 
Jeepack #: This is the only part that I keep struggling with. There's probably something really basic I'm not thinking of. Lets say it's 3:00:00 and I want to get bar 0 and 1 from EURUSD M1 and know if bar 0's duration lapsed but I can only do one array query...What do I do?

I'm assuming you're using either CopyRates, CopyClose or CopyOpen with datetime start_pos and either int count, but I haven't figured out exactly how to do it yet, I'm thinking about it right now. I bet it's something pretty simple so I'll be laughing when I find it. I didn't know I had this issue until very recently and I haven't given this option any thought yet. Am I on the right path?

Remember what I wrote — I calculate things incrementally and I always keep track of the previous state of a symbol.

Lets say the new bar event has occurred at roughly 03h00 (H1 bars). I have in my current state for the symbol the complete data related to the bar that opened 01h00 and closed at 02h00. So, at 03h00, I read the last 2 bars (current & previous) with "CopyRates( _Symbol, _Period, 0, 2, ... )". I then check the open time of the last/current bar from the rates copied. Is it 03h00? If yes, great — then the previous bar is the one I want, which is the bar 02h00-03h00.

Now, what if the last/current bar is not 03h00 but instead 02h00? That would mean that no new tick has arrived to update the state. So I take the current bar and consider that the bar for 02h00-03h00 as closed and store it, but mark it as "uncertain" in my state machine for the symbol. On the next new bar event, when I ran the CopyRates() again, I can update the bar's data, but 99% of the time there would have been no change to the data as the next tick would be for the open of the next bar. On the off chance when due to network latency the missing ticks were actually for that same bar and not the open of the next, then the data will be slightly different and require a recalculation of indicator values, but the difference will be so small and insignificant that it would not affect the decisions previously made, and even if the change was larger, that is part of the strategy rules, and part of the acceptable tolerance and uncertainty of those rules, that will be managed according with the trade and risk management.

Now, for the cases of missing bars. When I have do CopyRates() and the bar is behind the expected value by a full bar, then I store a "blank/missing" bar with the appropriate state of missing and the data as open = high = low = close = previous bar's close, and tick_volume = 0, real_volume = 0, spread = 0. On the next bar event I continue the process as described above. If i discover that due the network/broker problems, I have missed an entire bar, then I update the data as described before, and accept that uncertainty, but I log the situation in the log for analysis. If I lost data due to network/broker problems, then I have to solve the network/broker problem. There is nothing that can be done for the EA data after the fact. The decisions by the EA are correct based on the collected data at the time, and those are part of the strategy rules. I simply accept that. Trade and risk management will take care of the rest.

 
Jeepack #: This is the only part that I keep struggling with. There's probably something really basic I'm not thinking of. ****EDIT: oh wait, I just figured it out, I just don't usually use MqlRates so I hadn't realized that CopyRates isn't the same as CopyBuffer, now things are starting to make sense. It's as if you were querying a multi-dimensional array in a way. After you've copied it, you can then access things like the Open, Close, and bar open time from the data you previously copied. I didn't realize that, this is why I didn't understand how you had no synchronization issue.

I answered your previous version of the post before you edited it. Yes, correct, CopyRates is what I use and gets everything in one go.

 
Fernando Carreiro #:

I answered your previous version of the post before you edited it. Yes, correct, CopyRates is what I use and gets everything in one go.

Thanks for sharing this, everything you do to check for latency issues, missing bars, etc is really interesting! I did my best to write code that would replicate the fundamentals of what you do and I finally succeeded. I worked hard at it so it turns out I actually used the same method as you previously described but before I read your post :) copying two bars, checking index 1 first to see if it's the new bar I wanted, and if not, checking the index 0 bar instead. It was a really tough exercise so after all your help, I can say I really improved my EA coding skills.

I even took the time to create a successful OnTimer/OnTick combo like you suggested and the code is really good I think. It detects when the timeframe's duration lapses, and uses either OnTimer or sometimes OnTick if OnTick is faster than the timer. I made it so that I can adjust the interval I want updates to take place on with ENUM_TIMEFRAMES and the EA pretty much always picks up rates in the first 500 ms after a new time period's duration lapses without waiting for a new bar to form. Then I collect the new rate incrementally from rates[0] or rates[1] depending on if the new bar has formed or not.

I hope I was a good student!

This is when it runs with M1:

2022.04.29 01:32:00.294 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05319 | New bar open time: 2022.04.29 08:31:00

2022.04.29 01:33:00.453 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05303 | New bar open time: 2022.04.29 08:32:00

2022.04.29 01:34:00.400 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05299 | New bar open time: 2022.04.29 08:33:00

2022.04.29 01:35:00.329 Timer (EURUSD,M1) New event: OnTick | New bar close price: 1.05312 | New bar open time: 2022.04.29 08:34:00

2022.04.29 01:36:00.503 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05336 | New bar open time: 2022.04.29 08:35:00

2022.04.29 01:37:00.438 Timer (EURUSD,M1) New event: OnTick | New bar close price: 1.05379 | New bar open time: 2022.04.29 08:36:00

2022.04.29 01:38:00.379 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05381 | New bar open time: 2022.04.29 08:37:00

2022.04.29 01:39:00.304 Timer (EURUSD,M1) New event: OnTick | New bar close price: 1.05381 | New bar open time: 2022.04.29 08:38:00

2022.04.29 01:40:00.484 Timer (EURUSD,M1) New event: OnTimer | New bar close price: 1.05363 | New bar open time: 2022.04.29 08:39:00

Depending on if I slow down the millisecond timer or speed it up I can make all events occur within 300 ms from when a bar's duration lapses, using mostly the timer event, or slow down the reaction time and let OnTick win more often, triggering events between 500 ms and 800 ms after a bar's duration lapses. It's really cool stuff that I never did before.

I guess when it comes to avoiding synchronisation problems when making the EA that fast still is important, so I'm glad I developped my technique because it works perfectly with indicator buffers but I agree that if I would be making calculations from rates only without using indicators, I wouldn't need to have any validation steps in my code. 100% agreed. I'm glad I know both techniques now and with a timer that makes it even better.

I hope I can return the favor some day! ;)

 
Jeepack #: Thanks for sharing this, everything you do to check for latency issues, missing bars, etc is really interesting! I did my best to write code that would replicate the fundamentals of what you do and I finally succeeded. ... I hope I can return the favor some day! ;)

You are welcome! Well done putting in the effort into accomplishing your goal. 👍

1234
New comment