ArraySort() error on multidimensional array

New comment
 

Greetings,


I'm studying MQL4 and MQL5 for a few years now (I'm not a professional programmer, I'm self educated), but I find hard to understand topics still. Recently, I wanted to store chart object data from a sqlite database to a multi-dimensional array in a project, to minimize disk I/O. As far as I know, I need to declare the array somewhat like this:

struct MarkerData
{
   ENUM_TIMEFRAMES timeframe;
   datetime timestamp;
   ENUM_MarkerOptions flag;
};

MarkerData MarkerArray[];

But when I try to work with the array, so like to search in it with ArrayBsearch, I need to sort it first:

   ArraySort(MarkerArray);

This way I get an error:

'ArraySort' - constant cannot be modified

At this point, there were no data transaction between the database and the array, the error showed up when I added the array to the code. I tried to comment out each and every data types, same error. I tried ArraySort with a one-dimensional integer type array, to test, all went well. So it is quite sure, the struct declaration, or the struct-based array causes the error.

It is also veird, that I cannot add a string to the struct with that syntax, I get a

struct MarkerData
{
   ENUM_TIMEFRAMES timeframe;
   string objectname;
   datetime timestamp;
   ENUM_MarkerOptions flag;
};

'MarkerArray' - structures or classes containing objects are not allowed

built-in: bool ArraySort(void&)


I'm quite new to arrays (used a few here and there, but mostly one-dimensional), and never used structs before. What do I do wrong? The documentation is not clear, and I did not find relevant topics neither on the web, nor on youtube. Please help.

EDIT: the code was written in MQL5, not sure if there are any differences in the relevant sections between the 2 language variants.

SQLite: Native handling of SQL databases in MQL5
SQLite: Native handling of SQL databases in MQL5
  • www.mql5.com
The development of trading strategies is associated with handling large amounts of data. Now, you are able to work with databases using SQL queries based on SQLite directly in MQL5. An important feature of this engine is that the entire database is placed in a single file located on a user's PC.
 
J.P.Satrio:

Greetings,


I'm studying MQL4 and MQL5 for a few years now (I'm not a professional programmer, I'm self educated), but I find hard to understand topics still. Recently, I wanted to store chart object data from a sqlite database to a multi-dimensional array in a project, to minimize disk I/O. As far as I know, I need to declare the array somewhat like this:

But when I try to work with the array, so like to search in it with ArrayBsearch, I need to sort it first:

This way I get an error:

'ArraySort' - constant cannot be modified

At this point, there were no data transaction between the database and the array, the error showed up when I added the array to the code. I tried to comment out each and every data types, same error. I tried ArraySort with a one-dimensional integer type array, to test, all went well. So it is quite sure, the struct declaration, or the struct-based array causes the error.

It is also veird, that I cannot add a string to the struct with that syntax, I get a

'MarkerArray' - structures or classes containing objects are not allowed

built-in: bool ArraySort(void&)


I'm quite new to arrays (used a few here and there, but mostly one-dimensional), and never used structs before. What do I do wrong? The documentation is not clear, and I did not find relevant topics neither on the web, nor on youtube. Please help.

EDIT: the code was written in MQL5, not sure if there are any differences in the relevant sections between the 2 language variants.

There is, less known obviously, following detail about arrays, documented here:


What you will find on the lower end of the page is a table with functions, which can be considered "methods" for arrays.

There is a method called ArraySort, which takes in a pointer to a function for performing the sorting.

Same goes for the ArraySearch function.

Honestly, I have never used them before and therefore have no idea about the syntax.

But they exist, because you cannot perform ArraySort like you are trying, with complex objects as elements in your array.

This is a "limitation", and it is by definition of the structure or class. The main problem is, when you want to sort an array with structures, which member to use for the sorting? And how to tell the function about that?


 
Dominik Egert #:
There is, less known obviously, following detail about arrays, documented here:


What you will find on the lower end of the page is a table with functions, which can be considered "methods" for arrays.

There is a method called ArraySort, which takes in a pointer to a function for performing the sorting.

Same goes for the ArraySearch function.

Honestly, I have never used them before and therefore have no idea about the syntax.

But they exist, because you cannot perform ArraySort like you are trying, with complex objects as elements in your array.

This is a "limitation", and it is by definition of the structure or class. The main problem is, when you want to sort an array with structures, which member to use for the sorting? And how to tell the function about that?


The documentation reads:

"Sorts the values in the first dimension of a multidimensional numeric array in the ascending order."

So I thought it should work like:

MarkerArray[0]: Period_M1 ... 2023.08.15. 14:01 ... BUY

MarkerArray[1]: Period_M30 ... 2023.08.12. 12:00 ... SELL

MarkerArray[2]: Period_M5 ... 2023.08.17. 19:10 ... BUY

MarkerArray[3]: Period_M15 ... 2023.06.25. 19:45 ... BUY

MarkerArray[4]: Period_H1 ... 2023.05.15. 11:00 ... SELL


will go like:

MarkerArray[0]: Period_H1 ... 2023.05.15. 11:00 ... SELL

MarkerArray[1]: Period_M1 ... 2023.08.15. 14:01 ... BUY

MarkerArray[2]: Period_M15 ... 2023.06.25. 19:45 ... BUY

MarkerArray[3]: Period_M30 ... 2023.08.12. 12:00 ... SELL

MarkerArray[4]:  Period_M5 ... 2023.08.17. 19:10 ... BUY


When I tried the array.Sort() I got some different errors, but I didn't pay close attention, since I was thinking about the limitations you mentioned.

I don't really care about the order, I just wanted to verify existence of a certain datetime in the array, and ArrayBsearch requires the array to be sorted. But in that case, if the array with a struct format has such limitations I should approach this task from a for loop perspective? Something like:

for(int i = 0; i == ArraySize(MarkerData); i++)
{
        if(MarkerData[i].timestamp == <DATETIME TO MATCH>)
                {
                        some operations
                }
}

?
Maybe, I sorted it out for myself, but will need to try it out (haven't done yet).

 
J.P.Satrio #:

The documentation reads:

"Sorts the values in the first dimension of a multidimensional numeric array in the ascending order."

So I thought it should work like:

MarkerArray[0]: Period_M1 ... 2023.08.15. 14:01 ... BUY

MarkerArray[1]: Period_M30 ... 2023.08.12. 12:00 ... SELL

MarkerArray[2]: Period_M5 ... 2023.08.17. 19:10 ... BUY

MarkerArray[3]: Period_M15 ... 2023.06.25. 19:45 ... BUY

MarkerArray[4]: Period_H1 ... 2023.05.15. 11:00 ... SELL


will go like:

MarkerArray[0]: Period_H1 ... 2023.05.15. 11:00 ... SELL

MarkerArray[1]: Period_M1 ... 2023.08.15. 14:01 ... BUY

MarkerArray[2]: Period_M15 ... 2023.06.25. 19:45 ... BUY

MarkerArray[3]: Period_M30 ... 2023.08.12. 12:00 ... SELL

MarkerArray[4]:  Period_M5 ... 2023.08.17. 19:10 ... BUY


When I tried the array.Sort() I got some different errors, but I didn't pay close attention, since I was thinking about the limitations you mentioned.

I don't really care about the order, I just wanted to verify existence of a certain datetime in the array, and ArrayBsearch requires the array to be sorted. But in that case, if the array with a struct format has such limitations I should approach this task from a for loop perspective? Something like:

?
Maybe, I sorted it out for myself, but will need to try it out (haven't done yet).


ArrayBsearch uses binary or also known as bubble search, which has time complexity of O(log n) while your for loop has O(n)

If your array is of small size, something around 100 to (maybe) 250 elements, a for loop is most probably faster than binary search. Beyond that, you will begin to get great delays.

The main limitation is, by what value should ArraySort actually sort your structures?

With array.ArraySort() you can pass in your own custom comparison function, which gives you the possibility to control the comparison.

Same goes for array search, as you can pass in the same comparison function.

Though I don't know the signature to the function you must provide, I would guess it's something like this:

const int compare(const my_struct& left, const my_struct& right)
{
    return(left.timestamp - right.timestamp); // or whatever your comparison might be...
}
You will now call sort on the array something like this:

array.Sort(compare);


EDIT:

There is another option, since the sorting algorithm provided by MQL is not the fastest, you could switch to this library here from code base. It is very good implemented and addresses your issue as well:


This author has a few other sorting algorithms on codebase, so if you feel you need some else, take a look at what he has got. His implementations are actually of high quality.
 
Dominik Egert #:

ArrayBsearch uses binary or also known as bubble search, which has time complexity of O(log n) while your for loop has O(n)

If your array is of small size, something around 100 to (maybe) 250 elements, a for loop is most probably faster than binary search. Beyond that, you will begin to get great delays.

The main limitation is, by what value should ArraySort actually sort your structures?

With array.ArraySort() you can pass in your own custom comparison function, which gives you the possibility to control the comparison.

Same goes for array search, as you can pass in the same comparison function.

Though I don't know the signature to the function you must provide, I would guess it's something like this:

You will now call sort on the array something like this:



EDIT:

There is another option, since the sorting algorithm provided by MQL is not the fastest, you could switch to this library here from code base. It is very good implemented and addresses your issue as well:


This author has a few other sorting algorithms on codebase, so if you feel you need some else, take a look at what he has got. His implementations are actually of high quality.
The goal is to create a buttonpress function, which should, if an object exist with the matching ENUM_TIMEFRAME and timestamp values, modify the object properties, otherwise create an object. My main problem is, I have no idea how could I express a proper pattern in a separate sort function, since I'm not a professional programmer. The first dimension in the array is an ENUM_TIMEFRAMES value, but I also tried to rearrange the fields, in hope of the ArraySort will work, but it didn't. Time complexity is an important aspect, since there might be over 1000 elements in the array after some time, so I might simply just drop the array approach, and will operate with database queries instead (the idea was to minimize disk I/O when adding elements, but it is worth to change back to the original plan due to the fact, that for me it seems to be esaier to express db queries instead of the sort on a struct array).
 
J.P.Satrio #:
The goal is to create a buttonpress function, which should, if an object exist with the matching ENUM_TIMEFRAME and timestamp values, modify the object properties, otherwise create an object. My main problem is, I have no idea how could I express a proper pattern in a separate sort function, since I'm not a professional programmer. The first dimension in the array is an ENUM_TIMEFRAMES value, but I also tried to rearrange the fields, in hope of the ArraySort will work, but it didn't. Time complexity is an important aspect, since there might be over 1000 elements in the array after some time, so I might simply just drop the array approach, and will operate with database queries instead (the idea was to minimize disk I/O when adding elements, but it is worth to change back to the original plan due to the fact, that for me it seems to be esaier to express db queries instead of the sort on a struct array).
OK.

Two options come to mind. First, use a tree structure, available in MQL STL, easy to use, fast and reliable. (Just got fixed with last release)

An SQlite in MQL can also be allocated as memory db, fast, easy to use, and can be flushed to disk, if needed. (Although I don't know if MQ already fixed the bug about hooking a file based DB into a memory based DB) but you can do the copy within MQL to a file based DB and vice versa.

Memory DBs are really fast, but nothing compared to a tree structure. And arrays beat the shit out of trees... Though each type has its drawbacks....

Make your choice...
 
J.P.Satrio #:
The goal is to create a buttonpress function, which should, if an object exist with the matching ENUM_TIMEFRAME and timestamp values, modify the object properties, otherwise create an object. My main problem is, I have no idea how could I express a proper pattern in a separate sort function, since I'm not a professional programmer. The first dimension in the array is an ENUM_TIMEFRAMES value, but I also tried to rearrange the fields, in hope of the ArraySort will work, but it didn't. Time complexity is an important aspect, since there might be over 1000 elements in the array after some time, so I might simply just drop the array approach, and will operate with database queries instead (the idea was to minimize disk I/O when adding elements, but it is worth to change back to the original plan due to the fact, that for me it seems to be esaier to express db queries instead of the sort on a struct array).

A database for 1000 objects ? Not a good idea.

Just use a hashmap with the timeframe/timestamp as a key. You will find a library in Include/Generic.

Documentation on MQL5: Standard Library / Generic Data Collections / CHashMap
Documentation on MQL5: Standard Library / Generic Data Collections / CHashMap
  • www.mql5.com
CHashMap - Generic Data Collections - Standard Library - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Alain Verleyen #:

A database for 1000 objects ? Not a good idea.

Just use a hashmap with the timeframe/timestamp as a key. You will find a library in Include/Generic.

That hashmap is not a good implementation, actually.

Although the tree is not very efficient, it is at least correctly implemented. This is not the case for the hash map/unordered map.

That's why I didn't suggest it.

But I guess for 1000 or even 10000 Elements, it will be "good enough". Though I am sure with 10k elements, tree would be a better choice.

Tree with up to 16k elements means max 13 lookup operations to find an element.

Collisions in hashmap will be more expensive, especially when you have lots of coming and going of entries.

 
Dominik Egert #:
That hashmap is not a good implementation, actually.

Although the tree is not very efficient, it is at least correctly implemented. This is not the case for the hash map/unordered map.

That's why I didn't suggest it.

But I guess for 1000 or even 10000 Elements, it will be "good enough". Though I am sure with 10k elements, tree would be a better choice.

Tree with up to 16k elements means max 13 lookup operations to find an element.

Collisions in hashmap will be more expensive, especially when you have lots of coming and going of entries.

Come on, people want things working and the hashmap from MQ is working fine. If he wants to use a tree, that's fine for me too. He could also use an array by the way, he just doesn't know how.

Don't go off-topic ;-)

 
Alain Verleyen #:

A database for 1000 objects ? Not a good idea.

Just use a hashmap with the timeframe/timestamp as a key. You will find a library in Include/Generic.

Why not a good idea?
 
J.P.Satrio #:
Why not a good idea?
Try it and let us know.
1234
New comment