Developing a Replay System — Market simulation (Part 02): First experiments (II)

Introduction

In the previous article "Developing a Replay System — Market simulation (Part 01): First experiments (I)", we have seen some limitations when trying to create an event system with a short execution time to generate an adequate market simulation. It became clear that it was impossible to get less than 10 milliseconds with this approach. In many cases, this time is quite low. However, if you study the files attached to the article, you will see that 10 milliseconds is not a low enough time period. Is there any other method that would allow us to achieve the desired time of 1 or 2 milliseconds?

Before considering anything connected with the use of time in the range of a few milliseconds, it's important to remind everyone that this is not an easy task. The fact is that the timer provided by the operating system itself cannot reach these values. Therefore, this is a big, if not GIANT, problem. In this article, I will try to answer this question and to show how you can try to solve it by going beyond the time limit set by the operating system. I know that many people think that a modern processor can perform billions of calculations per second. However, it is one thing when the processor performs calculations, and quite another question is whether all the processes inside the computer can cope with the required tasks. Please note that we are trying to use exclusively MQL5 for this, without using any external code or DLL. We are only using pure MQL5.

Planning

To verify this, we will have to make some changes to the methodology. If this process works out, we won't have to mess with the replay creation system again. We will focus on other questions that will help us conduct research or training using real tick values or simulated values. The way to assemble 1-minute bars remains the same. This will be the main focus of this article.

We are going to use a maximally generic approach, and the best way I have found is to use a client-server-like system. I already explained this technique in my earlier article "Developing a trading Expert Advisor from scratch (Part 16): Accessing data on the web (II)". In that article, I showed three ways to transmit information within MetaTrader 5. Here we will use one of these methods, namely SERVICE. So, Market Replay will become a MetaTrader 5 service.

You might be thinking that I am going to create everything from scratch. But why would I do such a thing? Basically, the system is already running, however it does not reach the desired 1 minute time. You might be asking: "Do you think that changing the system to a service will solve this problem?" In fact, simply replacing the system with a service will not solve our problem. But if we isolate the creation of 1-minute bars from the rest of the EA system from the very beginning, then we will have less work later on, because the EA itself will cause a slight delay in the execution of bar construction. I will explain the reasons for this later.

Do you understand now why we are using a service? It is much more practical than other methods discussed above. We will be able to control it in the same way as I explained in the article about how to exchange messages between an EA and a service: Developing a trading EA from scratch (Part 17): Accessing data on the web (III). But here we will not focus on generating this control, we only want the service to generate the bars that will be placed on the chart. To make things more interesting, we are going to use the platform in a more creative way, not just using an EA and a service.

Just as a reminder, in the last attempt to reduce the time, we got the following result:

This was the best time we had. Here we will crush this time right away. However, I don't want you to get totally attached to these values or to the tests shown here. This series of articles related to the creation of the replay/simulator system, is already at a much more advanced stage in which I changed some concepts several times in order to actually get the system to work as expected. Even if at this point it all seems to be adequate, deep down here I made some mistakes related to timing tests. Such errors, or misconceptions, are not easy to notice in such an early system. As this series of articles develops, you will notice that this timing-related issue is much more complex and that it involves much more than just getting the CPU and the MetaTrader 5 platform provide a certain amount of data on the chart so that you could have an immersion in the replay/simulator system.

So don't take everything you see here literally. Follow this series of articles because what we are going to do here is not simple or easy to do.

Implementation

Let's start by creating the foundation of our system. These include:

1. Service for creating 1-minute bars
2. Script to start the service
3. EA for simulation (this will be discussed later)

Defining the Market Replay Service

To properly work with the service, we need to update our C_Replay class. But these changes are very small, so we won't go into detail. Basically, these are the return codes. However, there is one point that is worth noting separately, as it implements something additional. The code is as follows:

#define macroGetMin(A)  (int)((A - (A - ((A % 3600) - (A % 60)))) / 60)
                int Event_OnTime(void)
                        {
                                bool isNew;
                                int mili;
                                static datetime _dt = 0;
                                
                                if (m_ReplayCount >= m_ArrayCount) return -1;
                                if (m_dt == 0)
                                {
                                        m_Rate[0].close = m_Rate[0].open =  m_Rate[0].high = m_Rate[0].low = m_ArrayInfoTicks[m_ReplayCount].Last;
                                        m_Rate[0].tick_volume = 0;
                                        m_Rate[0].time = m_ArrayInfoTicks[m_ReplayCount].dt - 60;
                                        CustomRatesUpdate(def_SymbolReplay, m_Rate, 1);
                                        _dt = TimeLocal();
                                }
                                isNew = m_dt != m_ArrayInfoTicks[m_ReplayCount].dt;
                                m_dt = (isNew ? m_ArrayInfoTicks[m_ReplayCount].dt : m_dt);
                                mili = m_ArrayInfoTicks[m_ReplayCount].milisec;
                                while (mili == m_ArrayInfoTicks[m_ReplayCount].milisec)
                                {
                                        m_Rate[0].close = m_ArrayInfoTicks[m_ReplayCount].Last;
                                        m_Rate[0].open = (isNew ? m_Rate[0].close : m_Rate[0].open);
                                        m_Rate[0].high = (isNew || (m_Rate[0].close > m_Rate[0].high) ? m_Rate[0].close : m_Rate[0].high);
                                        m_Rate[0].low = (isNew || (m_Rate[0].close < m_Rate[0].low) ? m_Rate[0].close : m_Rate[0].low);
                                        m_Rate[0].tick_volume = (isNew ? m_ArrayInfoTicks[m_ReplayCount].Vol : m_Rate[0].tick_volume + m_ArrayInfoTicks[m_ReplayCount].Vol);
                                        isNew = false;
                                        m_ReplayCount++;
                                }
                                m_Rate[0].time = m_dt;
                                CustomRatesUpdate(def_SymbolReplay, m_Rate, 1);
                                mili = (m_ArrayInfoTicks[m_ReplayCount].milisec < mili ? m_ArrayInfoTicks[m_ReplayCount].milisec + (1000 - mili) : m_ArrayInfoTicks[m_ReplayCount].milisec - mili);
                                if ((macroGetMin(m_dt) == 1) && (_dt > 0))
                                {
                                        Print("Elapsed time: ", TimeToString(TimeLocal() - _dt, TIME_SECONDS));
                                        _dt = 0;
                                }                               
                                return (mili < 0 ? 0 : mili);
                        };
#undef macroGetMin

The highlighted parts have been added to the source code of the C_Replay class. What we do is define the delay time, that is, we will use exactly the value obtained in the line, but in milliseconds. Do not forget that this time will not be exact, as it depends on some variables. However, we will try to keep it as close to 1 millisecond as possible.

With these changes in mind, let's look at the service code below:

#property service
#property copyright "Daniel Jose"
#property version   "1.00"
//+------------------------------------------------------------------+
#include <Market Replay\C_Replay.mqh>
//+------------------------------------------------------------------+
input string    user01 = "WINZ21_202110220900_202110221759"; //File with ticks
//+------------------------------------------------------------------+
C_Replay Replay;
//+------------------------------------------------------------------+
void OnStart()
{
        ulong t1;
        int delay = 3;
        
        if (!Replay.CreateSymbolReplay(user01)) return;
        Print("Waiting for permission to start replay ...");
        GlobalVariableTemp(def_GlobalVariable01);
        while (!GlobalVariableCheck(def_SymbolReplay)) Sleep(750);
        Print("Replay service started ...");
        t1 = GetTickCount64();
        while (GlobalVariableCheck(def_SymbolReplay))
        {
                if ((GetTickCount64() - t1) >= (uint)(delay))
                {
                        if ((delay = Replay.Event_OnTime()) < 0) break;
                        t1 = GetTickCount64();
                }
        }
        GlobalVariableDel(def_GlobalVariable01);
        Print("Replay service finished ...");
}
//+------------------------------------------------------------------+

The above code is responsible for creating the bars. By placing this code here, we make the replay system function independently: the operation of the MetaTrader 5 platform will hardly affect or be affected by it. So, we can work with other things related to the control system, analysis and simulation of the replay. But this will be done later on.

Now comes an interesting thing: pay attention that the highlighted parts gave the GetTickCount64 function. This will provide a system equivalent to what we saw in the previous article but with one advantage: here the resolution will drop to a time of 1 millisecond. This precision is not exact, it is approximate, but the level of approximation is very close to what the real market movement would be. This does not depend on the hardware you use. After all, you can even create a loop which would guarantee greater precision, but it would be quite laborious since this time it would depend on the hardware used.

The next thing to do is the following script. Here is its full code:

#property copyright "Daniel Jose"
#property version   "1.00"
//+------------------------------------------------------------------+
#include <Market Replay\C_Replay.mqh>
//+------------------------------------------------------------------+
C_Replay Replay;
//+------------------------------------------------------------------+
void OnStart()
{
        Print("Waiting for the Replay System ...");
        while((!GlobalVariableCheck(def_GlobalVariable01)) && (!IsStopped())) Sleep(500);
        if (IsStopped()) return;
        Replay.ViewReplay();
        GlobalVariableTemp(def_SymbolReplay);
        while ((!IsStopped()) && (GlobalVariableCheck(def_GlobalVariable01))) Sleep(500);
        GlobalVariableDel(def_SymbolReplay);
        Print("Replay Script finished...");
        Replay.CloseReplay();
}
//+------------------------------------------------------------------+

As you can see, both codes are quite simple. However, they communicate with each other through global variables supported by the platform. Thus, we have the following scheme:

These scheme will be maintained by the platform itself. If the script closes, the service will be stopped. If the service stops, then the symbol we are using to execute the replay system will stop receiving data. This makes it super simple and highly sustainable. Any improvements (both in the platform and in the hardware) are automatically reflected in the overall performance. This is not a miracle – it is all achieved due to the small latencies that occur during each operation performed by the service process. Only this will actually affect the system in general, we don't need to worry about the script or the WA that we will develop in the future. Any improvements will only affect the service.

To save you the trouble of testing the system, you can preview the result in the image below. So you, dear reader, won't have to wait a whole minute to see the result on your chart.

As you can see, the result is very close to ideal. The extra 9 seconds can be easily eliminated using the system settings. Ideally, the time should be less than 1 minute, which will make it easier to adjust things, since we will only need to add a delay to the system. It's easier to add latency than to reduce it.. But if you think that the system time cannot be reduced, let's take a closer look at this.

There is a point that produces a delay in the system, which is in the service. This point that will actually generate a delay is highlighted in the code below. But what if we make this line a comment? What will happen to the system?

        t1 = GetTickCount64();
        while (GlobalVariableCheck(def_SymbolReplay))
        {
// ...  COMMENT ...  if ((GetTickCount64() - t1) >= (uint)(delay))
                {
                        if ((delay = Replay.Event_OnTime()) < 0) break;
                        t1 = GetTickCount64();
                }
        }
        GlobalVariableDel(def_GlobalVariable01);

The highlighted line will no longer be executed. In this case, I will save you from the need to test the system locally and having to wait one minute again. The execution result is shown in the video below. You can watch it entirely or jump to the part where only the final result is shown. Feel free to make your choice.

That is, the biggest challenge is to properly generate a delay. But the small deviation in time of 1 minute for the bar to be created is not really a problem. Since even on a real account, we don't have the exact time, as there is latency in information transmission. This latency is quite small, but it still exists.

Maximum speed. Really?

Here we will make one last attempt to make the system operate in less than 1 minute.

When you look at millisecond values, you can notice that sometimes we have a variation of only 1 millisecond between one line and another. But we will be treating everything within the same second. So, we can make a small change to the code. We will add a loop inside it, which may make a very big difference to the overall system.

The changes are shown below:

#define macroGetMin(A)  (int)((A - (A - ((A % 3600) - (A % 60)))) / 60)
inline int Event_OnTime(void)
                        {
                                bool isNew;
                                int mili;
                                static datetime _dt = 0;
                                
                                if (m_ReplayCount >= m_ArrayCount) return -1;
                                if (m_dt == 0)
                                {
                                        m_Rate[0].close = m_Rate[0].open =  m_Rate[0].high = m_Rate[0].low = m_ArrayInfoTicks[m_ReplayCount].Last;
                                        m_Rate[0].tick_volume = 0;
                                        m_Rate[0].time = m_ArrayInfoTicks[m_ReplayCount].dt - 60;
                                        CustomRatesUpdate(def_SymbolReplay, m_Rate, 1);
                                        _dt = TimeLocal();
                                }
                                isNew = m_dt != m_ArrayInfoTicks[m_ReplayCount].dt;
                                m_dt = (isNew ? m_ArrayInfoTicks[m_ReplayCount].dt : m_dt);
                                mili = m_ArrayInfoTicks[m_ReplayCount].milisec;
                                do
                                {
                                        while (mili == m_ArrayInfoTicks[m_ReplayCount].milisec)
                                        {
                                                m_Rate[0].close = m_ArrayInfoTicks[m_ReplayCount].Last;
                                                m_Rate[0].open = (isNew ? m_Rate[0].close : m_Rate[0].open);
                                                m_Rate[0].high = (isNew || (m_Rate[0].close > m_Rate[0].high) ? m_Rate[0].close : m_Rate[0].high);
                                                m_Rate[0].low = (isNew || (m_Rate[0].close < m_Rate[0].low) ? m_Rate[0].close : m_Rate[0].low);
                                                m_Rate[0].tick_volume = (isNew ? m_ArrayInfoTicks[m_ReplayCount].Vol : m_Rate[0].tick_volume + m_ArrayInfoTicks[m_ReplayCount].Vol);
                                                isNew = false;
                                                m_ReplayCount++;
                                        }
                                        mili++;
                                }while (mili == m_ArrayInfoTicks[m_ReplayCount].milisec);
                                m_Rate[0].time = m_dt;
                                CustomRatesUpdate(def_SymbolReplay, m_Rate, 1);
                                mili = (m_ArrayInfoTicks[m_ReplayCount].milisec < mili ? m_ArrayInfoTicks[m_ReplayCount].milisec + (1000 - mili) : m_ArrayInfoTicks[m_ReplayCount].milisec - mili);
                                if ((macroGetMin(m_dt) == 1) && (_dt > 0))
                                {
                                        Print("Elapsed time: ", TimeToString(TimeLocal() - _dt, TIME_SECONDS));
                                        _dt = 0;
                                }                               
                                return (mili < 0 ? 0 : mili);
                        };
#undef macroGetMin

If you notice, we now have an outer loop that does this 1ms test. Since it is very difficult to make a correct adjustment within the system so that we would take advantage of using this single millisecond, maybe it is better to take it out of the play.

We've only made one change. You can see the result in the video below.

For those who want something even faster, look at the result:

I think that's enough. We now have the creation of a 1-minute bar below this time. We can make adjustments to reach the perfect time, adding delays to the system. But we will not do it because the idea is to have a system that would allow us to do simulated studies. Anything close to 1 minute is fine for training and practicing. It doesn't have to be something exact.

Conclusion

Now we have the basics of the Replay system we are creating, and we can move on to the next points. See that everything has been resolved only by using settings and functions present in the MQL5 language, which proves that it can actually do much more than many people think.

But please note that our work has only just begun. There is still a lot to be done.