Monitoring trading environment changes

In the previous section related to the OnTrade event, we mentioned that some trading strategy programming approaches may require you to take snapshots of the environment and compare them with each other over time. This is a common practice when using OnTrade but it can also be activated on a schedule, on every bar, or even tick. Our monitor classes that can read the properties of orders, deals, and positions lacked the ability to save the state. In this section, we will present one of the trading environment caching options.

The properties of all trading objects are divided by types into three groups: integer, real, and string. Each object class has its own groups (for example, for orders, integer properties are described in the ENUM_ORDER_PROPERTY_INTEGER enumeration, and for positions they are described in ENUM_POSITION_PROPERTY_INTEGER), but the essence of division is the same. Therefore, we will introduce the PROP_TYPE enumeration, with the help of which it will be possible to describe to which type an object property belongs. This generalization comes up naturally since the mechanisms for storing and processing properties of the same type should be the same, regardless of whether the property belongs to an order, position, or deal.

Arrays are the simplest way to store property values. Obviously, due to the presence of three base types, we will need three different arrays. Let's describe them inside a new class TradeState nested in MonitorInterface (TradeBaseMonitor.mqh).

The basic template MonitorInterface<I,D,S> forms the basis of all applied monitor classes (OrderMonitor, DealMonitor, PositionMonitor). Types I, D, and S here correspond to concrete enumerations of integer, real, and string properties.

It is quite logical to include the storage mechanism in the base monitor, especially since the created property cache will be filled with data by reading properties from the monitor object.

The entire TradeState class has been made public because its fields would need to be accessed from the parent monitor object (which is passed as a pointer to the constructor), and besides TradeState will be used only in the protected part of the monitor (they cannot be reached from the outside).

In order to fill three arrays with property values of three different types, you must first find out the distribution of properties by type and indexes in each particular array.

For each trading object type (orders, deals, and positions), the identifiers of the 3 corresponding enumerations with properties of different types do not intersect and form a continuous numbering. Let's demonstrate this.

In the Enumerations chapter, we considered the script ConversionEnum.mq5 which implements the process function to log all elements of a particular enumeration. That script examined the ENUM_APPLIED_PRICE enum. Now we can create a copy of the script and analyze the other three enumerations. For example, like this:

As a result of its execution, we get the following log. The left column contains the numbering inside the enumerations, and the values on the right (after the '=' sign) are the built-in constants (identifiers) of the elements.

For example, the property with a constant of 0 is a string POSITION_SYMBOL, the properties with constants 1 and 2 are integers POSITION_TIME and POSITION_TYPE, the property with a constant of 3 is a real POSITION_VOLUME, and so on.

Thus, constants are a system of end-to-end indexes on properties of all types, and we can use the same algorithm (based on EnumToArray.mqh) to get them.

For each property, you need to remember its type (which determines which of the three arrays will store the value) and the serial number among the properties of the same type (this will be the index of the element in the corresponding array). For example, we see that positions have only 3 string properties, so the strings array in the snapshot of one position will have to have the same size, and POSITION_SYMBOL (0), POSITION_COMMENT (11), and POSITION_EXTERNAL_ID (19) will be written to its indexes 0, 1, and 2.

The conversion of end-to-end indexes of properties into their type (one of PROP_TYPE) and into an ordinal number in an array of the corresponding type can be done once at the start of the program since enumerations with properties are constant (built into the system). We write the resulting indirect addressing table into a static two-dimensional indices array. Its size in the first dimension will be dynamically determined as the total number of properties (of all 3 types). We will write the size into the limit static variable. A couple of cells are allocated for the second dimension: indices[i][0] — type PROP_TYPE, indices[i][1] — index in one of the arrays ulongs, doubles, or strings (depending on indices[i][0]).

Variables j, d, and s will be used to sequentially index properties within each of the 3 different types. Here's how it's done in the static method calcIndices.

The boundary method returns the maximum constant among all elements of the given enumeration E.

The largest value of all three types of enumerations determines the range of integers that should be sorted in accordance with the property type to which they belong.

Here we use the detect method which returns true if the integer is an element of an enumeration.

The last question is how to run this calculation when the program starts. This is achieved by utilizing the static nature of the variables and the method.

Note that limit is initialized by the result of calling our calcIndices function.

Having a table with indexes, we implement the filling of arrays with property values in the cache method.

We loop through the entire range of properties from 0 to limit and, depending on the property type in indices[i][0], write its value to the element of the ulongs, doubles, or strings array under the number indices[i][1] (the corresponding element of the array is passed by reference to the _get method).

A call of owner.get(e, value) refers to one of the standard methods of the monitor class (here it is visible as an abstract pointer MonitorInterface). In particular, for positions in the PositionMonitor class, this will lead to PositionGetInteger, PositionGetDouble, or PositionGetString calls. The compiler will choose the correct type. Order and deal monitors have their own similar implementations, which are automatically included by this base code.

It is logical to inherit the description of a snapshot of one trading object from the monitor class. Since we have to cache orders, deals, and positions, it makes sense to make the new class a template and collect all common algorithms suitable for all objects in it. Let's call it TradeBaseState (fileTradeState.mqh).

One of the specific monitor classes described earlier is hidden under the letter M (OrderMonitor.mqh, PositionMonitor.mqh, DealMonitor.mqh). The basis is the state caching object of the newly introduced M::TradeState class. Depending on M, a specific index table will be formed inside (one for class M) and arrays of properties will be distributed (own for each instance of M, that is, for each order, deal, position).

The cached variable contains a sign of whether the arrays in the state are filled with property values, and whether to query properties on an object to return values from the cache. This will be required later to compare the saved and current states.

In other words, when cached is set to false, the object will behave like a regular monitor, reading properties from the trading environment. When cached equals true, the object will return previously stored values from internal arrays.

By default, caching is, of course, enabled.

We must also provide a method that directly performs caching (filling arrays). To do this, just call the cache method for the state object.

What is the refresh method?

So far, we have been using monitor objects in simple mode: creating, reading properties, and deleting them. At the same time, property reading assumes that the corresponding order, deal, or position was selected in the trading context (inside the constructor). Since we are now improving monitors to support the internal state, it is necessary to ensure that the desired element is re-allocated in order to read the properties even after an indefinite time (of course, with a check that the element still exists). To implement this, we have added the refresh virtual method to the template MonitorInterface class.

It must return true upon successful allocation of an order, deal, or position. If the result is false, one of the following errors should be contained in the built-in _LastError variable:

• 4753 ERR_TRADE_POSITION_NOT_FOUND;
• 4754 ERR_TRADE_ORDER_NOT_FOUND;
• 4755 ERR_TRADE_DEAL_NOT_FOUND;

In this case, the ready member variable, which signals the availability of the object, must be reset to false in implementations of this method in derived classes.

For example, in the PositionMonitor constructor, we had and still have such an initialization. The situation is similar to order and deal monitors.

Now we will add the refresh method to all specific classes of this kind (see example PositionMonitor):

But populating cache arrays with property values is only half the battle. The second half is to compare these values with the actual state of the order, deal, or position.

To identify differences and write indexes of changed properties to the changes array, the generated TradeBaseState class provides the getChanges method. The method returns true when changes are detected.

As you can see, the main work is entrusted to a certain method diff in class M. This is a new method: we need to write it. Fortunately, thanks to OOP, you can do this once in the base template MonitorInterface, and the method will appear immediately for orders, deals, and positions.

So, everything is ready to form specific caching classes for orders, deals, and positions. For example, positions will be stored in the extended monitor PositionState on the base of PositionMonitor.

Similarly, a caching class for deals is defined in the file TradeState.mqh.

With orders, things are a little more complicated, because they can be active and historical. So far we have had one generic monitor class for orders, OrderMonitor. It tries to find the submitted order ticket both among the active orders and in the history. This approach is not suitable for caching, because Expert Advisors need to track the transition of an order from one state to another.

For this reason, we add 2 more specific classes to the OrderMonitor.mqh file: ActiveOrderMonitor and HistoryOrderMonitor.

Each of them searches for a ticket only in their area. Based on these monitors, you can already create caching classes.

The final touch that we will add to the TradeBaseState class for convenience is a special method for converting a property value to a string. Although there are several versions of the stringify methods in the monitor, they will all "print" either values from the cache (if the member variable cached equals true) or values from the original object of the trading environment (if cached equals false). To visualize the differences between the cache and the changed object (when these differences are found), we need to simultaneously read the value from the cache and bypass the cache. In this regard, we add the stringifyRaw method which always works with the property directly (due to the fact that the cached variable is temporarily reset and reinstalled).

Let's check the performance of the caching monitor using a simple example of an Expert Advisor that monitors the status of an active order (OrderSnapshot.mq5). Later we will develop this idea for caching any set of orders, deals, or positions, that is, we will create a full-fledged cache.

The Expert Advisor will try to find the last one in the list of active orders and create the OrderState object for it. If there are no orders, the user will be prompted to create an order or open a position (the latter is associated with placing and executing an order on the market). As soon as an order is found, we check if the order state has changed. This check is performed in the OnTrade handler. The Expert Advisor will continue to monitor this order until it is unloaded.

In addition to displaying an array of changed properties, it would be nice to display the changes themselves. Therefore, instead of an ellipsis, we will add such a fragment (it will be useful to us in future classes of full-fledged caches).

Here we use the new stringifyRaw method. After displaying the changes, do not forget to update the cache state.

If you run the Expert Advisor on an account with no active orders and place a new one, you will see the following entries in the log (here buy limit for EURUSD is created below the current market price).

Here you can see how the status of the order changed from STARTED to PLACED. If, instead of a pending order, we opened on the market with a small volume, we might not have time to receive these changes, because such orders, as a rule, are set very quickly, and their observed status changes from STARTED immediately to FILLED. And the latter already means that the order has been moved to history. Therefore, parallel history monitoring is required to track them. We will show this in the next example.

Please note that there may be many OnTrade events but not all of them are related to our order.

Let's try to set the Take Profit level and check the log.

Next, change the expiration date: from GTC to one day.

Here, in the process of changing our order, the price had enough time to change, and therefore we "hooked" an intermediate notification about the new value in ORDER_PRICE_CURRENT. And only after that, the expected changes in ORDER_TYPE_TIME and ORDER_TIME_EXPIRATION got into the log.

Next, we removed the order.

Now for any actions with the account that lead to OnTrade events, our Expert Advisor will output TRADE_ORDER_NOT_FOUND, because it is designed to track a single order. If the Expert Advisor is restarted, it will "catch" another order if there is one. But we will stop the Expert Advisor and start preparing for a more urgent task.

As a rule, caching and controlling changes is required not for a single order or position, but for all or a set of them, selected according to certain conditions. For these purposes, we will develop a base template class TradeCache (TradeCache.mqh) and, based on it, we will create applied classes for lists of orders, deals, and positions.

In this template, the letter T denotes one of the classes of the TradeState family. As you can see, an array of such objects in the form of auto-pointers is reserved under the name data.

The letter F describes the type of one of the filter classes (OrderFilter.mqh, including HistoryOrderFilter, DealFilter.mqh, PositionFilter.mqh) used to select cached items. In the simplest case, when the filter does not contain let conditions, all elements will be cached (with respect to sampling history for objects from history).

The letter E corresponds to the enumeration in which the property identifying the objects is located. Since this property is usually SOME_TICKET, the enumeration is assumed to be an integer ENUM_SOMETHING_PROPERTY_INTEGER.

The NOT_FOUND_ERROR variable is intended for the error code that occurs when trying to allocate a non-existent object for reading, for example, ERR_TRADE_POSITION_NOT_FOUND for positions.

In parameters, the main class method scan receives a reference to the configured filter (it should be configured by the calling code).

At the beginning of the method, we collect the identifiers of already cached objects into the tickets array. Obviously, on the first run, it will be empty.

Next, we fill the objects array with tickets of relevant objects using a filter. For each new ticket, we create a caching monitor object T and add it to the data array. For old objects, we analyze the presence of changes by calling data[j][].getChanges(changes) and then update the cache by calling data[j][].update().

As you can see, in each phase of the change, that is, when an object is added or after it is changed, the onAdded and onUpdated methods are called. These are virtual stub methods that the scan can use to notify the program of the appropriate events. Application code is expected to implement a derived class with overridden versions of these methods. We will touch on this issue a little later, but for now, we will continue to consider the method scan.

In the above loop, all found tickets in the tickets array are set to zero, and therefore the remaining elements correspond to the missing objects of the trading environment. Next, they are checked by calling getChanges and comparing the error code with NOT_FOUND_ERROR. If this is true, the onRemoved virtual method is called. It returns a boolean flag (provided by your application code) saying whether the item should be removed from the cache.

At the very end of the scan method, the data array is cleared of null elements but this fragment is omitted here for brevity.

The base class provides standard implementations of the onAdded, onRemoved, and onUpdated methods which display the essence of events in the log. By defining the PRINT_DETAILS macro in your code before including the header file TradeCache.mqh, you can order a printout of all the properties of each new object.

We will not present the onUpdated method, because it practically repeats the code for outputting changes from the Expert Advisor OrderSnapshot.mq5 shown above.

Of course, the base class has facilities for getting the size of the cache and accessing a specific object by number.

Based on the base TradeCache class, we can easily create certain classes for caching lists of positions, active orders, and orders from history. Deal caching is left as an independent task.

To summarize the process of developing the presented functionality, we present a diagram of the main classes. This is a simplified version of UML diagrams which can be useful when designing complex programs in MQL5.

Class diagram of monitors, filters, and caches of trading objects

Templates are marked in yellow, abstract classes are left in white, and certain implementations are shown in color. Solid arrows with filled tips indicate inheritance, and dotted arrows with hollow tips indicate template typing. Dotted arrows with open tips indicate the use of the specified methods of each other by classes. Connections with diamonds are a composition (inclusion of some objects into others).

As an example of using the cache, let's create an Expert Advisor TradeSnapshot.mq5, which will respond to any changes in the trading environment from the OnTrade handler. For filtering and caching, the code describes 6 objects, 2 (filter and cache) for each type of element: positions, active orders, and historical orders.

No conditions are set for filters through the let method calls so that all discovered online objects will get into the cache. There is an additional setting for orders from the history.

Optionally, at startup, you can load past orders to the cache at a given history depth. This can be done via the HistoryLookup input variable. In this variable, you can select the last day, last week (by duration, not calendar), month (30 days), or year (360 days). By default, the past history is not loaded (more precisely, it is loaded only in 1 second). Since the macro PRINT_DETAILS is defined in the Expert Advisor, be careful with accounts with a large history: they can generate a large log if the period is not limited.

In the OnInit handler, we reset the caches (in case the Expert Advisor is restarted with new parameters), calculate the start date of the history in the origin variable, and call OnTrade for the first time.

The OnTrade handler is minimalistic as all the complexities are now hidden inside the classes.

Immediately after launching the Expert Advisor on a clean account, we will see the following message:

Let's try to execute the simplest test case: let's buy or sell on an "empty" account which has no open positions and pending orders. The log will record the following events (occurring almost instantly).

First, an active order will be detected.

Then this order will be moved to the history (at the same time, at least the status, execution time and position ID will change).

Note that these modifications happened within the same call of OnTrade. In other words, while our program was analyzing the properties of the new order (by calling orders.scan), the order was processed by the terminal in parallel, and by the time the history was checked (by calling history.scan), it has already gone down in history. That is why it is listed both here and there according to the last line of this log fragment. This behavior is normal for multithreaded programs and should be taken into account when designing them. But it doesn't always have to be. Here we are simply drawing attention to it. When executing an MQL program quickly, this situation usually does not occur.

If we were to check the history first, and then the online orders, then at the first stage we could find that the order is not yet in the history, and at the second stage that the order is no longer online. That is, it could theoretically get lost for a moment. A more realistic situation is to skip an order in its active phase due to history synchronization, i.e., right away fix it for the first time in history.

Recall that MQL5 does not allow you to synchronize the trading environment as a whole, but only in parts:

• Among active orders, information is relevant for the order for which the OrderSelect or OrderGetTicket function has just been called
• Among the positions, the information is relevant for the position for which the function PositionSelect, PositionSelectByTicket, or PositionGetTicket has just been called
• For orders and transactions in the history, information is available in the context of the last call of HistorySelect, HistorySelectByPosition, HistoryOrderSelect, HistoryDealSelect

In addition, let's remind you that trade events (like any MQL5 events) are messages about changes that have occurred, placed in the queue, and retrieved from the queue in a delayed way, and not immediately at the time of the changes. Moreover, the OnTrade event occurs after the relevant OnTradeTransaction events.

Try different program configurations, debug, and generate detailed logs to choose the most reliable algorithm for your trading system.

Let's return to our log. On the next triggering of OnTrade, the situation has already been fixed: the cache of active orders has detected the deletion of the order. Along the way, the position cache saw an open position.

After some time, we close the position. Since in our code the position cache is checked first (positions.scan), changes to the closed position are included in the log.

Further in the same call of OnTrade, we detect the appearance of a closing order and its instantaneous transfer to history (again, due to its fast parallel processing by the terminal).

There are already 2 orders in the history cache, but the position and active order caches that were analyzed before the history cache have not yet applied these changes.

But in the next OnTrade event, we see that the position is closed, and the market order has disappeared.

If we monitor caches on every tick (or once per second, but not only for OnTrade events), we will see changes in the ORDER_PRICE_CURRENT and POSITION_PRICE_CURRENT properties on the go. POSITION_PROFIT will also change.

Our classes do not have persistence, that is, they live only in RAM and do not know how to save and restore their state in any long-term storage, such as files. This means that the program may miss a change that happened between terminal sessions. If you need such functionality, you should implement it yourself. In the future, in Part 7 of the book, we will look at the built-in SQLite database support in MQL5, which provides the most efficient and convenient way to store the trading environment cache and similar tabular data.