Transferring calendar database to tester

The calendar is available for MQL programs only online, and therefore testing news trading strategies poses some difficulties. One of the solutions is to independently create a certain image of the calendar, that is, the cache, and then use it inside the tester. Cache storage technologies can be different, such as files or an embedded SQLite database. In this section, we will show an implementation using a file.

In any case, when using the calendar cache, remember that it corresponds to a specific point in time X. In all "old" events (financial reports) that happened before X, actual values are already set, and in later ones (in "future", relative to X) there are no actual values, and will not be until a new, more recent copy of the cache appears. In other words, it makes no sense to test indicators and Expert Advisors to the right of X. As for those to the left of X, you should avoid looking ahead, that is, do not read the current indicators until the time of publication of each specific news.

Attention! When requesting calendar data in the terminal, the time of all events is reported taking into account the current time zone of the server, including a possible correction for "daylight saving" time (as a rule, this means increasing the timestamps by 1 hour). This synchronizes news releases with online quote times. However, past clock changes (half a year, a year ago, or more) are displayed only in quotes, but not in calendar events. The entire calendar database is read through MQL5 according to the server's current time zone. Because of this, any created calendar archive will contain the correct timestamps for those events that occurred with the same DST mode (on or off) that was active at the time of storing. For events in "opposite" half-years, it is required to independently make an adjustment for an hour after reading the archive. In the examples below, this situation is omitted.

Let's call the cache class CalendarCache and put it in a file named CalendarCache.mqh. We will need to save all 3 tables of the calendar base in the file (MqlCalendarCountry, MqlCalendarEvent, MqlCalendarValue). MQL5 provides functions FileWriteArray and FileReadArray (see Writing and reading arrays) that can directly write and read arrays of simple structures to files. However, 2 out of 3 structures in our case are not simple, because they have string fields. Therefore, we need a mechanism for separately storing strings, similar to the one we already used in the CalendarFilter class (there was an array of strings stringCache, and the index of the desired string from this array was indicated in the filters).

In order to avoid missing strings from different "calendar" structures in one "dictionary", we will prepare a template class StringRef: the type parameter T will be any of MqlCalendar structures. This will give us a separate string cache for countries, and a separate string cache for event types.

The strings are stored in the cache array by using operator=, and extracted from it using operator[] (with a dummy index that is always omitted). Each object stores only the index of the string in the array. The cache array is declared static, so it will accumulate all string fields of one T structure. Those who wish can change the method of caching in such a way that each field of the structure has its own array, but this is not important for us.

Writing an array to a file and reading from a file are performed by a pair of static methods save and load: both take a file handle as a parameter.

Taking into account the StringRef class, let's describe structures that duplicate the standard calendar structures which use StringRef objects instead of string fields. For example, for MqlCalendarCountry we get MqlCalendarCountryRef. Standard and modified structures are copied into each other in a similar way by overloaded operators '=' and '[]'.

Note that the assignment operators of the first method have the overload '=' from StringRef, due to which all the lines fall into the array StringRef<MqlCalendarCountry>::cache. In the second method, the '[]' operator calls invisibly get the address of the string and return from StringRef directly the string stored at that address in the cache array.

The MqlCalendarEventRef structure is defined in a similar way, but only 3 fields in it (source_url, event_code, name) require replacing type string by StringRef<MqlCalendarEvent>. The MqlCalendarValue structure does not require such transformations, since there are no string fields in it.

This concludes the preparatory stages, and you can proceed to the main cache class CalendarCache.

From general considerations, as well as for compatibility with the already developed CalendarFilter class, let's describe the fields in the cache that specify the context (country or currency), the range of dates for stored events, and the moment of cache generation (time X, variable t).

Actually, it does not make much sense to set restrictions when creating a cache from a calendar. A full cache is probably more practical since its size is not critical as it is about two dozens of megabytes till the middle of 2022 (this include historical data from 2007 with events planned until 2024). However, restrictions can be useful for demo programs with artificially reduced functionality.

It is obvious that arrays of calendar structures should be provided in the cache to store all the data.

Initially, they are filled from the calendar database by the update method.

The t field is a sign of cache health, with the time of filling arrays.

The filled cache object can be written to a file using the save method. At the beginning of the file, there is a header CALENDAR_CACHE_HEADER — this is the string "MQL5 Calendar Cache\r\nv.1.0\r\n", which allows you to make sure that the format is correct when reading. Next, the method saves the context, from, to, and t variables, as well as the values array, "as is". Before the array itself, we write down its size in order to restore it when reading.

With arrays events and countries come our wrapper structures with the "Ref" suffix. The helper method store converts the events array into an array of simple structures erefs, in which strings are replaced by numbers in the dictionary of strings StringRef<MqlCalendarEvent>. Such simple structures can already be written to a file in the usual way, but for their subsequent reading, it is also necessary to save all the lines of the dictionary (calling StringRef<MqlCalendarEvent> ::save(handle)). Country structures are converted and saved to file in the same way.

The aforementioned store method is quite simple: in it, in a loop over the elements, an overloaded assignment operator is executed in the MqlCalendarEventRef or MqlCalendarCountryRef structures.

To load the received file into the cache object, a mirror method load is written. It reads data from the file into variables and arrays in the same order, simultaneously performing reverse transformations of string fields for event types and countries.

Helper method restore uses the overload of the '[]' operator in a loop over elements in the MqlCalendarEventRef or MqlCalendarCountryRef structures to get the line itself by line number and assign it to a standard MqlCalendarEvent or MqlCalendarCountry structure.

At this stage, we already could write a simple test indicator based on the CalendarCache class, run it on an online chart, and save it to a file with the calendar cache. Then the file could be loaded from the copy of the indicator in the tester, and the full set of events could be received. However, this is not enough for practical developments.

The fact is that for quick access to data, it is required to provide indexing, a well-known concept in programming, which we will touch on later, in the chapter on databases. In theory, we could use the built-in SQLite engine to store the cache, and then we would get indexes "for free", but more on that later.

The point of indexing is easy to understand if we imagine how to effectively implement analogs of standard calendar functions in our cache. For example, the event ID is passed in the CalendarValueById function. Direct enumeration of records in the array values would be very time-consuming. Therefore, it is required to supplement the array with some "data structure" that would allow us to optimize the search. "Data structure" is in quotation marks, because it is not about the meaning of the programming language (struct), but in general about the architecture of data construction. It can consist of different parts and be based on different organizational principles. Of course, the extra data will require memory, but exchanging memory for speed is a common approach in programming.

The simplest solution for indexing is a separate two-dimensional array, sorted in ascending order so that it can be quickly searched using the ArrayBsearch function. Two elements are enough for the second dimension: values with indices [i][0], by which sorting is performed, contain identifiers, and values [i][1] contain the ordinal positions in the array of structures.

Another frequently used concept is hashing which is a transformation of the initial values into some keys (hashes, integers) in such a way that it provides the minimum number of collisions (matches of keys for different initial data). The fundamental property of keys is a close to uniform random distribution of their values, due to which they can be used as indexes in pre-allocated arrays. Computing a hash function for a single element of the original data is a fast process that actually yields the address of the element itself. For example, the well-known hash map data structures follow this principle.  

If the two original values do get the same hash (although this is rare), they are lined up in a list for their key, and a sequential search will be performed within the list. However, since the hash functions are chosen so that the number of matches is small, the search usually hits the target as soon as the hash is computed.

For demonstration, we will use both approaches in the CalendarCache class: hashing and binary search.

The MetaTrader 5 package includes a set of classes for creating hash maps (MQL5/Include/Generic/HashMap.mqh), but we will manage with our own simpler implementation, in which only the principle of using the hash function remains.

Scheme for indexing data by hashing

In our case, it is enough to hash only the identifiers of the calendar objects. The hashing function that we choose will have to convert the identifier to an index inside a special array: the position of the identifier in the array of "calendar" structures will be stored in a cell with this index. For countries, types of events, and specific news, it is allocated according to its own array.

Their elements will store the sequence number of the entry in the relevant array (countries, events, values).

For each of the "redirect" arrays, at least 2 times more elements should be allocated than the number of corresponding structures in the database (and in the cache) of the calendar. Due to this redundancy, we minimize the number of hash collisions. It is believed that the greatest efficiency is achieved when choosing a size equal to a prime number. Therefore, the class has a static method size2prime which returns the recommended size of the array of hash "baskets" (one of id4 -arrays) according to the number of elements in the source data.

The whole process of calendar hashing is described in the hash method. Let's look at its beginning using the example of an array of structures countries, and the other two arrays are treated similarly.

So we get the recommended "plain" index size id4country from the size of the countries array by calling size2prime. Initially, the index array is filled with the value -1, that is, all its elements are free. Further in the loop through the countries, it is necessary to calculate the hash for each next country identifier and using it find a free index in the id4country array. This is the job for the helper method place.

The hash function inside place is the expression (MathSwap(id) ^ 0xEFCDAB8967452301) % n, where id is our identifier, and n is the size of the index array. Thus, the result of calculations is always reduced to a valid index inside array[]. The principle of choosing a hash function is a separate topic that is beyond the scope of the book.

If the cell at the p positionin the index array is not occupied (equal to -1), we immediately write the location address of the calendar structure to the element [p]. If the cell is already occupied, we try to select the next one using the formula p = (p + attempt) % n, where attempt is a counter of attempts (this is our camouflaged version of the list of elements with a matched hash). If the number of failed attempts reaches one-tenth of the original data, indexing will fail, but this is practically impossible with our oversized index array size and the known nature of the hashed data (unique identifiers).

As a result of hashing the array of structures, we get a filled index array (there are free spaces in it, but this is how it is intended), through which we can find the location of the corresponding structure in the array of structures by the identifier of the calendar element. This is done by the find method which is opposite in meaning to place.

Let's show how it is used in practice. The standard calendar functions include CalendarCountryById and CalendarEventById. When you need to test an MQL program in the tester, it will not be able to directly access them, but it will be able to load the calendar cache into the CalendarCache object and therefore it should have similar methods.

They use the find method and index arrays id4country and id4event.

But these are not the most desired features of the calendar. Much more often, an MQL program with a news strategy needs functions CalendarValueHistory, CalendarValueHistoryByEvent, CalendarValueLast, or CalendarValueLastByEvent. They provide quick access to calendar entries by time, country, or currency.

So, the CalendarCache class should provide similar methods. Here we will use the second method of "indexing" — through a binary search in a sorted array.

To implement the above methods, let's add 4 more two-dimensional arrays to the class to establish a correspondence between news and event type, news and country, news, and currency, as well as news and the time of its publication.

In the first element of each row, i.e., under the indices [i][0] an event ID, country, currency or time, respectively, will be recorded. In the second element of the series, under indices [i][1] IDs of specific news will be placed. After filling all the arrays once, they are sorted using ArraySort on values [i][0]. Then we can search by ID, for example, by event_id, for all such news in the value2event array: the ArrayBsearch function will return the number of the first matching element, followed by others with the same event_id until a distinct identifier is encountered. The order in the second "column" is not defined (can be any).

Quick search for related structures based on sorting

This operation of mutual binding of structures of different types is carried out in the bind method. The size of each "binding" array is the same as the size of the news array. Going through all the news in a loop, we use ready-made index arrays and the find method for fast addressing.

In the case of currencies, a special number obtained from the string using the currencyId function is taken as an identifier.

Now we can finally present the entire constructor of the CalendarCache class.

When launched on an online chart, the created object with default parameters will collect all calendar information (update), index it (hash), and link the tables (bind). If something goes wrong at any of the stages, the error sign will be 0 in the variable t. If successful, the value from the function TimeTradeServer will remain there (remember, it is placed inside update). Such a ready-to-use object can be exported to a file using the save method described above.

When launched in the tester, the object should be created with a special combination of parameters from and to (from > to) — in this case, the program will consider the context string a filename and will load the calendar state from it. The easiest way to do it is this:

Inside the method load we will also call hash and bind to bring the object into a working state.

Using the CalendarValueLast function as an example, we show an equivalent implementation of the calendarValueLast method (with exactly the same prototype). The cache will use the current "server" time as a change identifier, in the absence of an open software API for reading the online calendar change table. Hypothetically, we could use the information about the change IDs saved by the CalendarChangeSaver.mq5 service, but this approach requires a long-term collection of statistics before testing can begin. Therefore, the "server" time generated by the tester is accepted as a fairly adequate replacement.

When the MQL program requests changes for the first time with a null identifier, we simply return the value from TimeTradeServer.

If the change identifier is already non-zero, we continue the main branch of the algorithm.

Depending on the contents of the code and currency parameters, we find the identifiers of the country and currency. By default, it is 0, which means it searches for all changes.

Further along, using the transmitted time count change as the beginning of the search, we find all the news in value2time up to the new, current value TimeTradeServer. Inside the loop, we use the find method to look for the index of the corresponding MqlCalendarValue structure in the values array and, if necessary, compare the country and currency of the associated event type with the desired ones. All news items that meet the criteria are written to the result output array.

Methods calendarValueHistory, calendarValueHistoryByEvent, and calendarValueLastByEvent are implemented according to a similar principle (the latter actually delegates all the work to the method calendarValueLast discussed earlier). The complete source code can be found in the attached file CalendarCache.mqh.

Based on the cache class, it is logical to create a derived class CalendarFilter, which, when processing requests, would access the cache instead of the calendar.

The finished solution is in the file CalendarFilterCached.mqh. Due to the fact that the cache API was designed on the basis of the standard API, the integration is reduced to only forwarding filter calls to the cache object (autopointer cache).

To test the calendar in the tester, let's create a new version of the indicator CalendarMonitor.mq5 – CalendarMonitorCached.mq5.

The main differences are as follows.

We assume that some cache file will be created or already created under the name "xyz.cal" (in the folder MQL5/Files) and therefore connect it to the MQL program with the directive tester_file.

This directive ensures the transfer of the cache to any agents, including distributed ones (which, however, is more relevant for Expert Advisors, rather than an indicator). A cache file with this (or another name) can be created using a new input variable CalendarCacheFile. If the user changes the default name to something else, then to work in the tester, you will need to correct the directive (requires recompilation!), or transfer the file to the shared folder of terminals (this feature is supported in the cache class, but "left behind the scenes"), however, such a file is no longer available to remote agents.

The CalendarFilter object is now described as an autopointer, because depending on where the indicator is run, it can use the original class CalendarFilter as well as the derived class CalendarFilterCached.

At the beginning of OnInit, there is a new fragment that is responsible for generating the cache and reading it.

If the cache file has been read, we will get the finished object CalendarCache, which is passed to the CalendarFilterCached constructor. Otherwise, the program checks whether it is running in the tester or online. The absence of a cache in the tester is a fatal case. On a regular chart, the program creates a new object based on the built-in calendar data and saves it in the cache under the specified name. But if the file name is made empty, the indicator will work exactly as the original one — directly with the calendar.

Let's run the indicator on the EURUSD chart. The user will be warned that the specified file was not found and an attempt was made to save it. Provided that the calendar is enabled in the terminal settings, we should get approximately the following lines in the log. Below is a version with detailed diagnostic information. The details can be disabled by commenting out the directive in the source code #define LOGGING.

Now we can choose the indicator CalendarMonitorCached.mq5 in the tester and see in dynamics, based on history, how the news table changes.

News indicator with calendar cache in the tester

The presence of the calendar cache allows you to test trading strategies on the news. We will show this in the next section.