technical question about mql5 ArraySetAsSeries() function

[Imre Sopp]

I use SOA to distribute processed data . So, to make life easier i tried to use the ArraySetAsSeries() function for reverse access. For some reason ArraySetOfSeries() does not reverse my indexing order, I don't know if I'm doing something wrong or is it not supposed to work only with OnCalculate Copy_Rates data distribution? In this case it is still possible to use the mql5 ArraySetAsSeries() function or do I have to implement a solution like a reversed index iterator myself? 

Sorry for such a stupid question, but once when i worked with OnCalculate it worked like clockwork... I can't find any limitations of use the ArraySetAsSeries() in the documentation exclude static- and multidimensional arrays... So, my arrays are dynamic.

Thanks in advance!

[Marteo Gonzales Cosme]

Imre Sopp:

I use SOA to distribute processed data . So, to make life easier i tried to use the ArraySetAsSeries() function for reverse access. For some reason ArraySetOfSeries() does not reverse my indexing order, I don't know if I'm doing something wrong or is it not supposed to work only with OnCalculate Copy_Rates data distribution? In this case it is still possible to use the mql5 ArraySetAsSeries() function or do I have to implement a solution like a reversed index iterator myself? 

Sorry for such a stupid question, but once when i worked with OnCalculate it worked like clockwork... I can't find any limitations of use the ArraySetAsSeries() in the documentation exclude static- and multidimensional arrays... So, my arrays are dynamic.

Thanks in advance!

 ArraySetAsSeries() doesn't reverse the order of data and it only tells MQL5 how to interpret indexing -- as if index 0 holds the most recent data (right-to-left, descending order). 

I think ArrayReverse() function is what need if you need to reverse the elements of your array. (https://www.mql5.com/en/docs/array/array_reverse).

[Imre Sopp]

Marteo Gonzales Cosme #:

 ArraySetAsSeries() doesn't reverse the order of data and it only tells MQL5 how to interpret indexing -- as if index 0 holds the most recent data (right-to-left, descending order). 

I think ArrayReverse() function is what need if you need to reverse the elements of your array. (https://www.mql5.com/en/docs/array/array_reverse).

Thank you for answering, although unfortunately that wasn't the answer to my question.

As i said: For some reason ArraySetOfSeries() does not reverse my indexing order,   and: i tried to use the ArraySetAsSeries() function for reverse access 

Here I make it clear that I don't want to change the positions of the elements themselves but the indexing order of logical layer. I looked into the ArrayReverse() function already as a first alternative but documentation says, ArrayReverse() changes the positions of the elements themselves and therefore it is too expensive to run , especially considering that this is real time data that can potentially be updated every minute. So, that would be the worst choice in this situation...   Now let's think about whether there might be some explanatory reasons why ArraySetOfSeries() doesn't work ?

[Marteo Gonzales Cosme]

Imre Sopp #:

As i said: For some reason ArraySetOfSeries() does not reverse my indexing order,   and: i tried to use the ArraySetAsSeries() function for reverse access 

Can you show the partial code here and add comments what exactly your trying to achieve.  

[Imre Sopp]

ok I'll give some examples... we declare structure instances:

S_ZigZag_Engine_Live<double>              ZZ_Engine_Fast; 
S_ZigZag_Engine_Live<double>              ZZ_Engine_Medium;

in structure itself:

template<typename T> 
struct S_ZigZag_Engine_Live
{
   
   public:
   //  datapoints
   int  mem_reserve;
   T    arr      [];
   Time time     [];
   int  raw_i    [];
   //  x deltas
   int  x_bars   [];

in structure, there are also member functions:

void manage_datasize(int size, int reserve)
   {
        ArrayResize(arr,    size, reserve);
        ArrayResize(time,   size, reserve);
        ArrayResize(raw_i,  size, reserve);
        ArrayResize(x_bars, size, reserve);
   }

   void Reverse_Array_Indexes(bool reverse) 
   {
      ArraySetAsSeries(arr,    reverse);
      ArraySetAsSeries(time,   reverse);
      ArraySetAsSeries(raw_i,  reverse);
      ArraySetAsSeries(x_bars, reverse);
   }

I fill the arrays in natural order, starting from left 0 to right newest element and this indexing does not change. No matter where I call the Reverse_Array_Indexes() function... or directly set ArraySetAsSeries(arr, true); still nothing changes... I don't know if I've overworked or gone crazy or it's not supposed to work for custom distributed datasets

for example if i call it OnInit() scope:

int OnInit()
{
   ZZ_Engine_Fast.Reverse_Array_Indexes(true);
   ZZ_Engine_Medium.Reverse_Array_Indexes(true);
}

I probably have no problem building a logic layer that presents or distributes the indexes in reverse order. But the question is rather whether ArraySetAsSeries() should work in such a situation or not? 

but I would first trust the mql5 developers and would like to use the ready-made function, because they have probably built it more efficiently using maybe C languages and direct access to hardware. Therefore, the original function may be more efficient than my mql5 development, it depends... but yes, could someone please explain more specifically what theoretically could be causing this situation? Maybe it adds some meaning when I mention that all the data is distributed using customized methods, that maybe ArraySetAsSeries needs metadata that is collected using the classic Rates_Total, maybe they are connected in some way, could this be the reason?

[Stanislav Korotky]

Indexing is a logical stuff - it's alwais from a start (say, 0) to the end (say, N-1).

If you want to change direction of access by the indexes, you need to switch the ArraySetAsSeries flag between writing and reading.

[Imre Sopp]

Stanislav Korotky #:

Indexing is a logical stuff - it's alwais from a start (say, 0) to the end (say, N-1).

If you want to change direction of access by the indexes, you need to switch the ArraySetAsSeries flag between writing and reading.

Yes, that seems authentic. Thanks for the answer, I appreciate it.  

This immediately raises a number of additional questions. Indexing is a abstraction layer of course which maps the real physical memory address and logical position. Therefore I don't fully understand why it doesn't guarantee continued reverse access of array elements integrated as the lower level logical layer in cooperation with the dynamic array function itself as a permanent layer when reverse flag is true or how this function exactly work...

It's also possible that my head is just overheated and I need a little cooling :D Or it is possible to get the source code of the built-in mql5 functions somewhere, probably not?

[Imre Sopp]

Still seems that ArraySetAsSeries function only affects index interpretation for indicator buffers and time-series arrays. MetaTrader doesn't track memory layout of custom arrays and no internal mapping is created for non-buffer arrays?

[Conor Mcnamara]

Imre Sopp #:
Still seems that ArraySetAsSeries function only affects index interpretation for indicator buffers and time-series arrays. MetaTrader doesn't track memory layout of custom arrays and no internal mapping is created for non-buffer arrays?

Basically yes. Internally, the data is stored the same way. 

Simply: Index 0 always refers to the latest bar (most recent data) when ArraySetAsSeries(arr, true) . Index 0 refers to the oldest bar (earliest data) when ArraySetAsSeries(arr, false)   -- default in MQL5.

You will get the same output if your calculation loop is a decrementing loop or incrementing loop. If your loop is incrementing when buffers are set as series, your start index should be zero. If your loop is decrementing when buffers are set as series, your end index should be zero.

[Conor Mcnamara]

If you want to actually reverse arrays, you have ArrayReverse function

https://www.mql5.com/en/docs/array/array_reverse

[Imre Sopp]

Conor Mcnamara #:

If you want to actually reverse arrays, you have ArrayReverse function

Conor Mcnamara #:
Basically yes. Internally, the data is stored the same way. 

You're apparently talking again about MQL5 rates_total data distribution, which is designed to make it easier to work with indicator buffers and time series and where you probably get a automatically updated reversed view thanks to synchronized metadata?  (I don't remember if this is the case, but it seems, because the ArraySetAsSeries is called often even in the OnInit scope in the code and in codes that use the indicator buffer, it is not called after each update...).

But I’m referring to the default distribution using classical left-to-right array layout (from index 0 to N),  from scratch, without any metadata. Of course, in both cases, the original order of the data collecting is the same because we cannot collect data in reverse, we can only transmit or access it in reverse.

When dealing with default distributed data processing, the only way to access array elements in reverse order using ArraySetAsSeries(true) is to call it before reading. So, after adding each new element, if access is needed... between writing and reading as Stanislav Korotky has already pointed out.

Since data can only be collected in order from 0 to N, but when we need to access the newest element first, using ArrayReverse() would require physically rewriting the entire array every time a new element is added. As mentioned earlier, this approach is completely disaster for stream data processing. ArrayReverse() is a solid function, but it’s designed for batch processing, where data updates are infrequent. Therefore real-time scenarios, a much better approach is to call ArraySetAsSeries(true) after each iterator(i) element addition. This is an authentic approach, but it can also be done differently.

There is another options to built own logic layer functions for mirrored access of array elements, or there is even more faster prerocessor macro implementation can be applied. Also i wrote a simple codes that provides reverse access that doesn't need to be called after each element is added. I added them to the codebase with a description, anyone who wants it can explore it further.... but for 2 days now its status is: Validation state: Not started ...