Share Class data from EA with [to] an Indicator - page 2

[Alain Verleyen]

Shephard Mukachi #:

Hi Paul,

I hope you're well.  I think I could possibly figure this out through testing, but given you're using the 'custom events' approach I wondered if you could help me.  I've been having a hard time synchronising multiple timeframes in one indicator.  There seems to be some acrobatics that must be peformed, or maybe I just lack adequate understanding, or both.

From my understanding, MT5 indicators are async - which is good in the caller does not have to wait for data to be loaded.  But this 'not waiting' presents an issues, if indicator is accessing HTF series, using CopySeries, CopyClose etc. - it does not always load.  Weekends are particularly worse when the OnCalculate is not fired when markets are closed.  I have seen Anatoli Kazharski' workaround, it's ugly.  I have not tested it, but I do not have a reason to believe it does not work.  Have you encountered issues regarding multiple timeframes in indicators?

I have used checks as provided by the documentation https://www.mql5.com/en/docs/series/timeseries_access, but theses methods do not work either.

So with your approach of using 'custom events'  what would happen if the indicator reloads, setting prev_calculated = 0?  When I think of that mechanism, especially if working with 2 timeframes, what if say D1 indicator resets/reloads, but the H1 Indicator does not?  This mean that they are out of sink?

Maybe, I'm just tired, or lack adequate understanding of MT5 indicators work.

Any ideas will be greatly appreciated.

Thanks,

Shep

What is "Anatoli Kazharski' workaround" ?

[Shephard Mukachi]

Hi Alain,

In this article https://www.mql5.com/en/articles/752 Anatoli uses the OnTimer and CopyDataOnCalculate.  Copying the OnCalculate parameters allows the OnTimer to call the OnCalculate function.  This is repeated until all data is synchronised.  Before synchronisation, the OnCalculate keeps returning prev_calculated, which is at it's first call value of zero (0).  This approach means that on weekends, even without market data, the indicator works, and it also synchronises across multiple timeframes and symbols.  Anatoli's example is a multiple symbol project, but It works with multiple timeframes as well.

I thought I was doing something wrong before I discovered Anatoli's approach.  I have developed high performance overlapped named pipes, and IOCP servers in C and C++, so very familiar with asynchronous data handling, but I found MT5 multiple timeframe and symbol indicators very challenging.  Maybe because I lack adequate knowledge and skill in MQL5, I don't know.

I find Anatoli's approach so against the grain.  In my mind, I see an indicator as a neat way of aligning series data, along with all the additional calculations.  It's easy access, easier to reason about data access and indexing.  The ability to visualise also allows great development experience in visual testing.  Also the additional Copy of series is extra memory.  I did RAM tests on my 32GB RAM laptop.  I created 1832 * 3 timeframes iCustom indicators with 8, 16, 32, 64, and 512.  My system can easy handle 64 buffers, but that's like the threshold. Beyond that RAM rises up really fast. If I added a 4th timeframe, I ran out of memory. So only 3 timeframes are a push.  This was without an EA doing any work with Copy.. functions which also require memory for the 1832 symbols.  So keeping an extraneous series would be unncessarily taxing and even prohibitive.

If you're wondering why I created 1832 * 3 timeframes iCustom handles, I wanted to see how many handles the platform can handle - I think it can do more than that, and ofcourse a limit test for the RAM usage.  My trading idea is to find just 3 patterns on all the available symbols.  I use 2 timeframes, one for context, usually D1, and one for entry, say H1. The entry rules are very strict, making it extremely accurate, but then appears infrequently.  However, by searching for this pattern accross all available symbols, I can maintain high win/loss ratio but can trade volumes.

When live, I use services.  I have MT5 services running as a worker pool.  Some produce signals, some handle order placenment and others handle positions.  I developed a high performance shared memory queue in C++ DLL, for communicating between service and it works impressively.

That's why I need to be able to use a single indicator per symbol, which performs analysis on the 2 timeframes, and do so reliably. The only real challenge I'm facing is the multiple timeseries synchronisation.

I'm open to any help, ideas, pointing in a direction.  Maybe others have similar 'crazy' ideas like me, I don't know.

Any help, ideas are greatly welcome.

Thanks,

Shep

[Alain Verleyen]

Shephard Mukachi #:

Hi Alain,

...

I'm open to any help, ideas, pointing in a direction.  Maybe others have similar 'crazy' ideas like me, I don't know.

Any help, ideas are greatly welcome.

Thanks,

Shep

Thanks for the detailed explanation, interesting.

This is repeated until all data is synchronised.

The issue with multi-symbols / multi-timeframes (it's the same issue for both) is you can't never be sure you will get all the data on the same event (OnCalculate). It's not specifically related to indicators, but it's certainly more annoying with them.

Anatoli way to collect and memorize series data is heavy and unneeded. His general approach is correct though but it can be fully dynamic : 1. Detect an error while collecting data, 2. Launch a timer (a custom event can eventually be used too), 3. Collect the needed data and call OnCalculate() by code, kill the timer when all is ok. There is no other way to have something generic that works in all situations, including when the market is closed, when the platform restart, etc...

This is a weakness in the platform which was probably not designed to build such indicators. So I had to build a framework to manage it and it's then doable to manage multi-symbols AND multi-timeframes data in all circumstances.

[Shephard Mukachi]

Thank you for a very informative response Alain. 

I understand the idea behind using the timer for while until all data is synchronised then killing the time. 

That's actually a brilliant idea.  Anatoli is very smart, I could not have thought of that idea, but I still find that too clunky.

So last night I decided to look at my challenge from a different perspective.  Instead of creating my Indicator on the lower Lower Timeframe then try to acquire Higher Timeframe series using Copy... functions, I tried creating the indicator on the Higher Timeframe, then get Lower Timeframe data. 

So the anaylysis is split into 'tick and bar'.  In the main cycle loop, I check if(rates_total > prev_calculated) - then I do the Higher Timeframe analysis here, set context etc, otherwise it's a tick, so I check if a I have a new Lower Timeframe bar. 

This works.  For my given design, it's the ideal solution.  On every tick, If I get a an M1 bar for example, I create a new bar, and store it in a stack (I use circular stacks) - this will give me time series bar access.  I can then caclulate anything I want and just use the stacks as time series storage.  It's neat.  I ran it in the Tester, and to my pleasant surprise it works, I ran it live and it's magical.  I just have to watch out that I'm actually getting a real Lower Timeframe bar - (true series start datetime check).

It is a slightly different way of looking at the anaylsis, but it has given me added benefits.  My strategy does a lot of the heavy analysis on Higher timeframes, and the lower timeframes are used primarily to pick the perfect entry.  I can have several strategies implemented in context (HTF) and I can implement many entry strategies on the Lower Timeframes - it's powerful and very flexible, and highly scalable too.

This year, once I have completed my current project, I want to develop a better Framework and one based around Indicators.  My current framework is heavily Epxert Advisor oriented. MQL5 has truly surprised me by some of it's capabilities.  Not perfect, but truly powerful.

Thanks for your time  and response, it's grately appreciated.  

Shep