Coding valuewhen() too short for an article - lol

New comment
 

Introduction - the problem

When developing an Expert Advisor, or an indicator, it is not uncommon that we want to find the value of a series x when series y last had a value z, or in other words when a condition was last satisfied.

This is so common that other platforms and languages have developed solutions, yet metatrader has not.

The following MT5 forum topics' metadata illustrate the problem:

Topic Date posted Last updated  MT4/MT5 Outcome
ValueWhen   28/05/2005  13/04/2006 - 1  MT4  Partial solution - 1
Barssince(), vauewhen()  30/09/2006  28/04/2009  MT4  Partial solution - 2
Valuewhen Indicator   30/04/2008  30/04/2008  MT4  None
How build up a function or way to repalce a way to mimic the function "Valuewhen" in metatrader 4  28/04/2009  29/04/2009 - 3  MT4  None - 3
Function equivalent to ValueWhen in MQL4?   27/03/2014  27/03/2014  MT4  None

Notes:

  1. In metastock-like functions: barssince(), vauewhen() but only a single value returned from barssince() for the last occurrence, which would be used as a parameter for valuewhen().

  2. Referencing 1

  3. Until my posting of the same approach outlined here

So despite the obvious early demand for this function there has been no definitive solution in MT4 or MT5.


Edit:

Following the moderator's action - below, the original topics are now unabvailable which is unfortunate since at least one had some half useful code for a 'barssince()' equivalent...


Comparison with other languages

Pine Script on TradingView has a function, valuewhen(condition, source, occurrence), which returns a series of results. The function can be overloaded to process results of source type bool, int, float and color. The condition is any boolean statement which clearly must be indexed the same as the source. Occurrence allows for other instances than the last (occurrence = 0) by passing vaues >= 1 for occurrence.

Python with Pandas has a one line method

//python script
//def valuewhen(condition, source, occurrence):
//    return source.reindex(condition[condition].index.shift(occurrence).reindex(source.index).ffill()

which does the same thing using pandas methods.

MetaQuotes has valuewhen(occurrence, condition, source) which does the same, as does Amibroker with ValueWhen(condition,source,occurrence).


Designing the function as a template/include

Since the requirement arises in both EAs and indicators, it makes sense to write a module to be used as an include.

The specifications are as follows:

  1. It should parameterise the series condition(), source(), and result(). 
  2. It should be capable of handling different type combinations with variations for overload including string data.
  3. It should be able to handle different boolean statements of relation.
  4. have two different variations which can handle condition types double and string. 
Other types can be explicitly coded for or can rely on the builtin typecasting of MT5; the potential loss of data however implicit in some typecasting will probably imply a need on most occasions to modify the code.

The condition() and source() arrays should be indexed as series ie from right to left - in contrast to my first version of the code referenced at 3 above.



Managing different operations of relation

The built in functions/methods of other platforms referenced above allow for easy definition of the relation operations defining the condition.

In metatrader we do not have this luxury so for that reason we have to resort to use of the switch operator to determine the relationship being tested, with the addition of a nonnegator boolean switch to reduce the number of cases.

Custom enum types are declared:

enum comparatord {equals=0, greaterthan=1, greaterthanorequal=2, between=3, betweenorequal=4};
enum comparators {sequals=0, contains=1, isempty=2};

The first set is for numeric comparisons, the second for string.

Others could be written to interogate array lengths etc.

In this example we are considering the double type.

One of the comparatord is passed along with a boolean nonnegator, in the parameters:

bool valuewhen(double &condition[], comparatord relationship, bool nonnegator, double uvalue, double &source[], int occurrence, double &result[], double lvalue=NULL)

resulting in the following truth table:

comparatord nonnegator = true
 nonnegator = false
equals condition[] == uvalue condition[] != uvalue
 greaterthan condition[]   > uvalue  condition[] <= uvalue
 greaterthanorequal  condition[]  >= uvalue  condition[]   < uvalue
 between  condition[]   > lvalue && condition[]  <  uvalue  condition[] <= lvalue || condition[] >= uvalue
 betweenorequal  condition[] >= lvalue && condition[] <= uvalue  condition[]   < lvalue || condition[]   > uvalue

The switch() statement is used in a loop iterating through the condition[] array which looks like this:

for(int i=ArraySize(condition)-1; i>=0; i--)
     {
      int z = -1;
      switch(relationship)
        {
         case 0:
            if((condition[i] == uvalue && nonnegator == true) || (condition[i] != uvalue && nonnegator == false))
               z=i;
            break;
         case 1:
            if((condition[i] > uvalue && nonnegator == true) || (condition[i] <= uvalue && nonnegator == false))
               z=i;
            break;
         case 2:
            if((condition[i] >= uvalue && nonnegator == true) || (condition[i] < uvalue && nonnegator == false))
               z=i;
            break;
         case 3:
            if((condition[i] < uvalue && nonnegator == true && condition[i] > lvalue) || (condition[i] >= uvalue && nonnegator == false && condition[i] <= lvalue))
               z=i;
            break;
         case 4:
            if((condition[i] == uvalue && nonnegator == true) || (condition[i] != uvalue && nonnegator == false))
               z=i;
            break;
        }
      if(z >-1)
        {
         ArrayResize(matchindex,ArraySize(matchindex)+1);
         matchindex[ArraySize(matchindex)-1]=z;
        }
     }

Storing and manipulating the required index values, using vector math to avoid one of our loops, retrieving the result

matchindex[] is used in the above loop to store an array of indexes where the required condition is satisfied.

Originally I was then looping through matchindex[] to create the complementary array of indexes.

The idea is to have a list of indexes to access the result from source[], and a list of indexes to determine from where the result[] array should be filled by the value from source. There is no faster way of accessing the values and writing the result[] array than going through the loop that follows the next piece of code, however the generation of the second index list should instinctively be quicker using vector math:

   vector vmatchindex;
   vmatchindex.Assign(matchindex); //copies the array matchindex[] into the vector vmatchindex
   vmatchindex = -1*(vmatchindex-ArraySize(condition)+1); //subtracts the values in our new vector from the size of the condition[] array to give the inverse index list
   
   ArrayRemove(matchindex,0,occurrence);//remove the positions in the array to shift it according to the value of occurrence
   vmatchindex.Resize(vmatchindex.Size()-occurrence); // remove the positions that fall off the end due to the shift

Finally, the function iterates through the shifted copy, obtaing the shifted value; the result[] array is progressively filled from the index position obtained from the shifted vector subtracted from the ArraySize of result with the source[] data obtained with the index from matchindex[], and the result[] finally reversed:

   for(int i = vmatchindex.Size()-1; i>=0; i--)
     {
      ArrayFill(result,vmatchindex[i],ArraySize(result)-vmatchindex[i],source[matchindex[i]]);
     }
   ArrayReverse(result);


NULL vs NaN

There are areas in the code where, in other languages, NULL would be used, although python/pandas uses NaN. However, the python/pandas solution has no use of either in the calling or coding of the function.

While MQL5 recognises NULL as an input, it is converted to 0.0 in any numeric variable. Given that 0.0 may be a valid result, it is necessary to find an alternative. Thankfully the answer is in the ability to initialise an array of doubles with the value "nan", explicitly typecast to the array type:

ArrayInitialize(result, double("nan"));

This is then returned as nan if not overwritten by the subsequent code.

Unfortunately this will not work with non double arrays, so in the attached code the result[] array is maintained as type double. This allows the correct and useful reporting of the NULL/NaN condition. The important point to remember is that source must be compatable with a double result. The type(s) of condition, uvalue and lvalue must only be compatable with eachother. 


Conclusion

This is my attempt to reproduce a useful function, requested and discussed over 17 years ago in the forum, and not discussed in terms of dedicated topics for over 8 years. It draws on the logic from the elegant python/pandas solution, and uses the new vector functionality in MT5.

It explores the way to use and return NaN - a subject poorly documented, in my opinion, hitherto.

The attached include file contains the functions and the global variables that need to be declared to use the code. Other variations to allow different combinations of parameter types are possible, as long as the warnings above are conbsidered.

There will certainly be a better way to do this - I am self taught and this is my first article, so be gentle in your criticism, but I hope you find it useful and would welcome comments.

 
Do not flood the forum by writing the same post in multiple (and very old) topics. They have been removed.
 

I actually only wrote twice. The topics are old but not been addressed by metatrader updates. I answered one topic then though to write an article but ir was not long enough so I posted it as a new topic and linked it in all the unsolved but yes old topics so that people would find what is, in my humble opinion, a solution. Whether the old topics are seen by the ops or newcomers is of no consequence, surely...


Of couse, if the admins pruned the site more effectively, there would not be the availability to reply to such old topics... but then neither would the longstanding need for a solution be exposed.


Several comments in more rrecent topics that do not have 'valuewhen' in the title are testament to the prolonged need...

New comment