Developing a Replay System — Market simulation (Part 22): FOREX (III)

Introduction

In the previous article, "Developing a Replay System — Market simulation (Part 21): FOREX (II), we mainly focused on solving system issues, especially those concerning the replay/simulation configuration file. However, since there is already a lot of information here for those who study and follow the articles of the system to learn how to create their own programs, we decided to end the article when the setup system was already in a working state in which we can work comfortably for quite a long time.

But if you have tested the attachment, you may have noticed that while the replay/simulation system works quite consistently in the stock market, the same cannot be said about the forex market. And here I don't just mean forex itself, but any asset that follows the same price representation concepts as forex, i.e. using bid as the underlying value.

Although this is the third article on this topic, I must explain for those who have not yet understood the difference between the stock market and the foreign exchange market: the big difference is that in the Forex there is no, or rather, we are not given information about some points that actually occurred during the course of trading. While this may seem like it only applies to the forex market, I don't mean it exclusively. This is because in the foreign exchange market we have our own method of obtaining certain information and the trading model is completely different from the stock market model. I think it's easier to make the distinction this way. Anything in these articles that relates to forex should be understood as applying to any type of market where the representation is done through the Bid value, as opposed to what happens in the stock market where the Last value is used.

Thus, by covering the foreign exchange market in the same way as the stock market (which is already covered by the market replay system), we will be able to replay or simulate any type of market. And one of the things that still gives us problems in the system is that if we disable the display of the creation of bars generated by replay or the simulator, the graph will look incorrect. We encountered this problem in the past, when we developing a system to cover the stock market. But since the display is different in the forex market and the basic price is Bid, the current system cannot handle this problem properly. If you try to move the studied position to another point when the bar display is turned off, all indicators will be incorrect, since there will be no bar between the old position and the new one. Therefore, we will start this article by fixing this problem.

Correcting the rapid positioning system

To solve this problem, we will have to make some changes to the code, not because it is incorrect, but because it cannot work with Bid plotting. To put it more precisely, the problem is that it is impractical to read and convert ticks into 1-minute bars (even in the C_FileTicks class) for subsequent use in the rapid positioning process. This is because we cannot represent price values based on Bid. To understand why this is so, let's look at the code that is responsible for performing this conversion. This code can be seen below:

inline bool ReadAllsTicks(const bool ToReplay)
    {
#define def_LIMIT (INT_MAX - 2)
#define def_Ticks m_Ticks.Info[m_Ticks.nTicks]

        string   szInfo;
        MqlRates rate;

        Print("Loading ticks for replay. Please wait...");
        ArrayResize(m_Ticks.Info, def_MaxSizeArray, def_MaxSizeArray);
        m_Ticks.ModePlot = PRICE_FOREX;
        while ((!FileIsEnding(m_File)) && (m_Ticks.nTicks < def_LIMIT) && (!_StopFlag))
        {
            ArrayResize(m_Ticks.Info, m_Ticks.nTicks + 1, def_MaxSizeArray);
            szInfo = FileReadString(m_File) + " " + FileReadString(m_File);
            def_Ticks.time = StringToTime(StringSubstr(szInfo, 0, 19));
            def_Ticks.time_msc = (def_Ticks.time * 1000) + (int)StringToInteger(StringSubstr(szInfo, 20, 3));
            def_Ticks.bid = StringToDouble(FileReadString(m_File));
            def_Ticks.ask = StringToDouble(FileReadString(m_File));
            def_Ticks.last = StringToDouble(FileReadString(m_File));
            def_Ticks.volume_real = StringToDouble(FileReadString(m_File));
            def_Ticks.flags = (uchar)StringToInteger(FileReadString(m_File));
            m_Ticks.ModePlot = (def_Ticks.volume_real > 0.0 ? PRICE_EXCHANGE : m_Ticks.ModePlot);
            if (def_Ticks.volume_real > 0.0)
            {
                ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
                m_Ticks.nRate += (BuiderBar1Min(rate, def_Ticks) ? 1 : 0);
                m_Ticks.Rate[m_Ticks.nRate] = rate;
            }
            m_Ticks.nTicks++;
        }
        FileClose(m_File);
        if (m_Ticks.nTicks == def_LIMIT)
        {
            Print("Too much data in the tick file.\nCannot continue...");
            return false;
        }
        return (!_StopFlag);
#undef def_Ticks
#undef def_LIMIT
    }

Please note that to call the function that creates bars, there must be a traded volume. This volume occurs only when the last price value changes. When using Bid prices, this volume will always be zero, that is, the subroutine is not called. The first thing we need to do is remove this code from here since we don't actually know if we will be working with Bid or Last plotting when reading ticks. So, the above function will be modified as below:

inline bool ReadAllsTicks(const bool ToReplay)
    {
#define def_LIMIT (INT_MAX - 2)
#define def_Ticks m_Ticks.Info[m_Ticks.nTicks]

        string   szInfo;
            
        Print("Loading ticks for replay. Please wait...");
        ArrayResize(m_Ticks.Info, def_MaxSizeArray, def_MaxSizeArray);
        m_Ticks.ModePlot = PRICE_FOREX;
        while ((!FileIsEnding(m_File)) && (m_Ticks.nTicks < def_LIMIT) && (!_StopFlag))
        {
            ArrayResize(m_Ticks.Info, m_Ticks.nTicks + 1, def_MaxSizeArray);
            szInfo = FileReadString(m_File) + " " + FileReadString(m_File);
            def_Ticks.time = StringToTime(StringSubstr(szInfo, 0, 19));
            def_Ticks.time_msc = (def_Ticks.time * 1000) + (int)StringToInteger(StringSubstr(szInfo, 20, 3));
            def_Ticks.bid = StringToDouble(FileReadString(m_File));
            def_Ticks.ask = StringToDouble(FileReadString(m_File));
            def_Ticks.last = StringToDouble(FileReadString(m_File));
            def_Ticks.volume_real = StringToDouble(FileReadString(m_File));
            def_Ticks.flags = (uchar)StringToInteger(FileReadString(m_File));
            m_Ticks.ModePlot = (def_Ticks.volume_real > 0.0 ? PRICE_EXCHANGE : m_Ticks.ModePlot);
            m_Ticks.nTicks++;
        }
        FileClose(m_File);
        if (m_Ticks.nTicks == def_LIMIT)
        {
            Print("Too much data in the tick file.\nCannot continue...");
            return false;
        }
        return (!_StopFlag);
#undef def_Ticks
#undef def_LIMIT
    }

Why am I doing this? The reason is that only after reading the entire file we will know whether the plotting will be done using the Bid or Last prices. Namely, this line guarantees this. However, it will only be valid after the file has been read. What will happen when we call the conversion of ticks to bars, what will be used at the display stage? For now, we are not discussing this. Please see the function below:

datetime LoadTicks(const string szFileNameCSV, const bool ToReplay = true)
    {
        int      MemNRates,
                 MemNTicks;
        datetime dtRet = TimeCurrent();
        MqlRates RatesLocal[];
        
        MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
        MemNTicks = m_Ticks.nTicks;
        if (!Open(szFileNameCSV)) return 0;
        if (!ReadAllsTicks(ToReplay)) return 0;
        if (!ToReplay)
        {
            ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
            ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
            CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
            dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
            m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
            m_Ticks.nTicks = MemNTicks;
            ArrayFree(RatesLocal);
        }else
        {
            CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_TRADE_CALC_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CALC_MODE_EXCH_STOCKS : SYMBOL_CALC_MODE_FOREX);
            CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_CHART_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CHART_MODE_LAST : SYMBOL_CHART_MODE_BID);
        }
        m_Ticks.bTickReal = true;
        
        return dtRet;
    };

We cannot call the tick conversion function anywhere in the above function. This is because at some points we will need to convert ticks to bars and we will use the real ticks file as if it were a file of previous 1-minute bars. This can be seen from the presence of the CustomRatesUpdate function in the above code. For this reason, we must call the conversion function before the call of CustomRatesUpdate. However, if you look at the conversion function, you will see that it is not suitable for use in the above code. The source function is shown below:

inline bool BuiderBar1Min(MqlRates &rate, const MqlTick &tick)
    {
        if (rate.time != macroRemoveSec(tick.time))
        {
            rate.real_volume = 0;
            rate.tick_volume = 0;
            rate.time = macroRemoveSec(tick.time);
            rate.open = rate.low = rate.high = rate.close = tick.last;
        
            return true;
        }
        rate.close = tick.last;
        rate.high = (rate.close > rate.high ? rate.close : rate.high);
        rate.low = (rate.close < rate.low ? rate.close : rate.low);
        rate.real_volume += (long) tick.volume_real;
        rate.tick_volume += (tick.last > 0 ? 1 : 0);

        return false;
    }

It is inconvenient to use this function where we need it. Therefore, we will have to create another function to perform the conversion. But besides this we have another problem. How do we know where to start the conversion? Remember that a call can occur at different times and we may have different needs at each time. In one of the calls we could download ticks for use in replay. In another call, we could load ticks that would have to be discarded soon after because they would be used as previous bars. As you can see, this needs to be carefully thought out so as not to reach a dead end.

All this will be much simpler and easier to do if you analyze the changes that need to be made and implement them exactly as planned. So we'll start by changing the calling procedure and then work on the conversion function. The new calling procedure is shown below:

datetime LoadTicks(const string szFileNameCSV, const bool ToReplay = true)
    {
        int      MemNRates,
                 MemNTicks;
        datetime dtRet = TimeCurrent();
        MqlRates RatesLocal[];
        
        MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
        MemNTicks = m_Ticks.nTicks;
        if (!Open(szFileNameCSV)) return 0;
        if (!ReadAllsTicks(ToReplay)) return 0;
        BuiderBar1Min(MemNTicks);
        if (!ToReplay)
        {
            ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
            ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
            CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
            dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
            m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
            m_Ticks.nTicks = MemNTicks;
            ArrayFree(RatesLocal);
        }else
        {
            CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_TRADE_CALC_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CALC_MODE_EXCH_STOCKS : SYMBOL_CALC_MODE_FOREX);
            CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_CHART_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CHART_MODE_LAST : SYMBOL_CHART_MODE_BID);
        }
        m_Ticks.bTickReal = true;
        
        return dtRet;
    };

Pay attention to how we solved the first part of our problem. By placing this line, we ensure that the ticks are converted to 1-minute bars before the possible call of CustomRatesUpdate. At the same time, we tell the conversion function where the conversion process should begin. We put the conversion function call here rather than in the read function in order to avoid adding an unnecessary variable to the read function. Here we have access to the variable which we need for determining the range in which we should work. Now we can move on to implementing the function that converts ticks into 1-minute bars. We will have to do this in a way that works for both the Forex type and the exchange type markets. Seems complicated, doesn't it? Again, if you don't plan out what you're going to do, you can get stuck in a coding cycle that will cause you to give up trying to implement the correct code.

I don't want to give you a solution here: I want you to learn to think in such a way that you can find a solution that works for you. Now let's think about the following: The original conversion procedure already allowed us to convert the last price to a 1-minute bar. We need to add a loop to this procedure so that it reads all the ticks read from the file. The starting point of this loop is passed to us by the calling program, and the end point will be the last tick read. So far everything is going well. But we also need to ensure that if the system detects that we are using plotting based on the Bid value, this value will replace the Last price that was originally used. This way we can convert the Bid value to a 1 minute bar without much effort. Interesting, isn't it? The implementation of this idea is shown below:

inline void BuiderBar1Min(const int iFirst)
    {
        MqlRates rate;
        double   dClose = 0;
        
        rate.time = 0;
        for (int c0 = iFirst; c0 < m_Ticks.nTicks; c0++)
        {
            switch (m_Ticks.ModePlot)
            {
                case PRICE_EXCHANGE:
                    if (m_Ticks.Info[c0].last == 0.0) continue;
                    dClose = m_Ticks.Info[c0].last;
                    break;
                case PRICE_FOREX:
                    dClose = (m_Ticks.Info[c0].bid > 0.0 ? m_Ticks.Info[c0].bid : dClose);
                    if (dClose == 0.0) continue;
                    break;
            }
            if (rate.time != macroRemoveSec(m_Ticks.Info[c0].time))
            {
                ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
                rate.time = macroRemoveSec(m_Ticks.Info[c0].time);
                rate.real_volume = 0;
                rate.tick_volume = 0;
                rate.open = rate.low = rate.high = rate.close = dClose;
            }else
            {
                rate.close = dClose;
                rate.high = (rate.close > rate.high ? rate.close : rate.high);
                rate.low = (rate.close < rate.low ? rate.close : rate.low);
                rate.real_volume += (long) m_Ticks.Info[c0].volume_real;
                rate.tick_volume++;
            }
            m_Ticks.Rate[(m_Ticks.nRate += (rate.tick_volume == 0 ? 1 : 0))] = rate;
        }
    }

The original function is highlighted in green to help you identify where it is present. These codes are a new part of the function. This part is entirely responsible for exchanging Last to Bid or vice versa so that the closing price is suitable to create a 1-minute bar. The loop I mentioned earlier is at this stage. Although this function solves most of our problems, it does not solve the tick volume problem if we use Bid type plotting. To solve this problem, we will have to slightly modify the previous function so that the final code looks like this:

inline void BuiderBar1Min(const int iFirst)
    {
        MqlRates rate;
        double  dClose = 0;
        bool    bNew;
        
        rate.time = 0;
        for (int c0 = iFirst; c0 < m_Ticks.nTicks; c0++)
        {
            switch (m_Ticks.ModePlot)
            {
                case PRICE_EXCHANGE:
                    if (m_Ticks.Info[c0].last == 0.0) continue;
                    dClose = m_Ticks.Info[c0].last;
                    break;
                case PRICE_FOREX:
                    dClose = (m_Ticks.Info[c0].bid > 0.0 ? m_Ticks.Info[c0].bid : dClose);
                    if ((dClose == 0.0) || (m_Ticks.Info[c0].bid == 0.0)) continue;
                    break;
            }
            if (bNew = (rate.time != macroRemoveSec(m_Ticks.Info[c0].time)))
            {
                ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
                rate.time = macroRemoveSec(m_Ticks.Info[c0].time);
                rate.real_volume = 0;
                rate.tick_volume = (m_Ticks.ModePlot == PRICE_FOREX ? 1 : 0);
                rate.open = rate.low = rate.high = rate.close = dClose;
            }else
            {
                rate.close = dClose;
                rate.high = (rate.close > rate.high ? rate.close : rate.high);
                rate.low = (rate.close < rate.low ? rate.close : rate.low);
                rate.real_volume += (long) m_Ticks.Info[c0].volume_real;
                rate.tick_volume++;
            }
            m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
        }
    }

In this code, which actually solves the tick volume problem, we had to add a new variable. Its value is determined at this stage, when we evaluate whether to add a new bar or not. This variable is used only here to let us know whether we will add a new bar to the system or not.

Please note the following: When we use the Last plotting mode, in order for the tick volume to be correct, we must start the counter from scratch. But when we use plotting by Bid, we must start the value from one. Otherwise, we will get incorrect data in tick volume.

The reason is in the details. But if you don't take the right precautions, they can harm you.

Correcting the tick volume

You might think there are no more errors in the system. However, there are still shortcomings that need to be addressed, and one of them is the volume problem. In the previous topic, we corrected the volume of ticks that the system reports in the event of a rapid change in the position we are analyzing. But if you run replay or simulation without quickly changing the position, the volume information will be incorrect.

It's not that the code is wrong, quite the opposite. If you are using a system to replay data or simulate an asset that uses the Last value for plotting, the reported tick volume will be correct. But if you use the Bid price, as happens in FOREX, this volume will be incorrect. We now need to solve this problem so that the volume information is correct. To understand what the problem is, let's look at the code that is responsible for performing this calculation:

inline void CreateBarInReplay(const bool bViewMetrics, const bool bViewTicks)
    {
#define def_Rate m_MountBar.Rate[0]

        bool    bNew;
        MqlTick tick[1];
        static double PointsPerTick = 0.0;

        if (bNew = (m_MountBar.memDT != macroRemoveSec(m_Ticks.Info[m_ReplayCount].time)))
        {
            PointsPerTick = (PointsPerTick == 0.0 ? SymbolInfoDouble(def_SymbolReplay, SYMBOL_TRADE_TICK_SIZE) : PointsPerTick);            
            if (bViewMetrics) Metrics();
            m_MountBar.memDT = (datetime) macroRemoveSec(m_Ticks.Info[m_ReplayCount].time);
            def_Rate.real_volume = 0;
            def_Rate.tick_volume = 0;
        }
        def_Rate.close = (m_Ticks.ModePlot == PRICE_EXCHANGE ? (m_Ticks.Info[m_ReplayCount].volume_real > 0.0 ? m_Ticks.Info[m_ReplayCount].last : def_Rate.close) :
               (m_Ticks.Info[m_ReplayCount].bid > 0.0 ? m_Ticks.Info[m_ReplayCount].bid : def_Rate.close));
        def_Rate.open = (bNew ? def_Rate.close : def_Rate.open);
        def_Rate.high = (bNew || (def_Rate.close > def_Rate.high) ? def_Rate.close : def_Rate.high);
        def_Rate.low = (bNew || (def_Rate.close < def_Rate.low) ? def_Rate.close : def_Rate.low);
        def_Rate.real_volume += (long) m_Ticks.Info[m_ReplayCount].volume_real;
        def_Rate.tick_volume += (m_Ticks.Info[m_ReplayCount].volume_real > 0 ? 1 : 0);
        def_Rate.time = m_MountBar.memDT;
        CustomRatesUpdate(def_SymbolReplay, m_MountBar.Rate);
        if (bViewTicks)
        {
            tick = m_Ticks.Info[m_ReplayCount];
            if (!m_Ticks.bTickReal)
            {
                static double BID, ASK;
                double dSpread;
                int    iRand = rand();
    
                dSpread = PointsPerTick + ((iRand > 29080) && (iRand < 32767) ? ((iRand & 1) == 1 ? PointsPerTick : 0 ) : 0 );
                if (tick[0].last > ASK)
                {
                    ASK = tick[0].ask = tick[0].last;
                    BID = tick[0].bid = tick[0].last - dSpread;
                }
                if (tick[0].last < BID)
                {
                    ASK = tick[0].ask = tick[0].last + dSpread;
                    BID = tick[0].bid = tick[0].last;
                }
            }
            CustomTicksAdd(def_SymbolReplay, tick); 
        }
        m_ReplayCount++;
        
#undef def_Rate
    }

In the case of an asset with the Bid-based representation, this calculation will have no effect, since this type of volume simply does not exist in forex. However, this calculation here will generate incorrect values for a Bid-based display system. This is due to the fact that in these cases the tick does not contain information about the traded volume. Thus, the tick volume value will always be zero.

However, in the calculation used to generate tick volume at the stage of creating 1-minute bars that will be used in case of fast movement, this calculation will give the correct value. Therefore, we must correct the above code. It will look like this:

inline void CreateBarInReplay(const bool bViewTicks)
    {
#define def_Rate m_MountBar.Rate[0]

        bool bNew;
        MqlTick tick[1];
        static double PointsPerTick = 0.0;

        if (bNew = (m_MountBar.memDT != macroRemoveSec(m_Ticks.Info[m_ReplayCount].time)))
        {
            PointsPerTick = (PointsPerTick == 0.0 ? SymbolInfoDouble(def_SymbolReplay, SYMBOL_TRADE_TICK_SIZE) : PointsPerTick);            
            m_MountBar.memDT = (datetime) macroRemoveSec(m_Ticks.Info[m_ReplayCount].time);
            if (m_Ticks.ModePlot == PRICE_FOREX) CustomRatesUpdate(def_SymbolReplay, m_MountBar.Rate, (def_Rate.time < m_MountBar.memDT ? 1 : 0));
            def_Rate.real_volume = 0;
            def_Rate.tick_volume = 0;
        }
        def_Rate.close = (m_Ticks.ModePlot == PRICE_EXCHANGE ? (m_Ticks.Info[m_ReplayCount].volume_real > 0.0 ? m_Ticks.Info[m_ReplayCount].last : def_Rate.close) :
                                                               (m_Ticks.Info[m_ReplayCount].bid > 0.0 ? m_Ticks.Info[m_ReplayCount].bid : def_Rate.close));
        def_Rate.open = (bNew ? def_Rate.close : def_Rate.open);
        def_Rate.high = (bNew || (def_Rate.close > def_Rate.high) ? def_Rate.close : def_Rate.high);
        def_Rate.low = (bNew || (def_Rate.close < def_Rate.low) ? def_Rate.close : def_Rate.low);
        def_Rate.real_volume += (long) m_Ticks.Info[m_ReplayCount].volume_real;
        def_Rate.tick_volume += ((m_Ticks.ModePlot == PRICE_FOREX) && (m_Ticks.Info[m_ReplayCount].bid > 0.0) ? 1 : (m_Ticks.Info[m_ReplayCount].volume_real > 0 ? 1 : 0));
        def_Rate.time = m_MountBar.memDT;
        CustomRatesUpdate(def_SymbolReplay, m_MountBar.Rate);
        if (bViewTicks)
        {
            tick = m_Ticks.Info[m_ReplayCount];
            if (!m_Ticks.bTickReal)
            {
                static double BID, ASK;
                double dSpread;
                int    iRand = rand();
    
                dSpread = PointsPerTick + ((iRand > 29080) && (iRand < 32767) ? ((iRand & 1) == 1 ? PointsPerTick : 0 ) : 0 );
                if (tick[0].last > ASK)
                {
                    ASK = tick[0].ask = tick[0].last;
                    BID = tick[0].bid = tick[0].last - dSpread;
                }
                if (tick[0].last < BID)
                {
                    ASK = tick[0].ask = tick[0].last + dSpread;
                    BID = tick[0].bid = tick[0].last;
                }
            }
            CustomTicksAdd(def_SymbolReplay, tick); 
        }
        m_ReplayCount++;
        
#undef def_Rate
    }

Don't ask me why, but for some strange reason that I personally have no idea about, we have to add this line here. If you don't add it, the value indicated in the tick volume will be incorrect. Pay attention that there is a condition in the function. This avoids problems when using the fast positioning system, and prevents the appearance of a strange bar that would be out of time on the system's chart. Although this is a very strange reason, everything else works as expected. This will be a new calculation where we will count ticks in the same way - both when working with a Bid-based asset and when working with the Last-based instrument.

As you may have noticed, this calculation is quite simple. But the interesting thing is that we have to send Rate values inside the custom asset a second time. After the bar closed. I could not understand the reason for this. Everything is so strange that this sending is only necessary if the Bid type is used, which is very interesting.

There's one more thing you probably noticed in the previous function. Now the metrics system no longer exists. In a sense, I was thinking about removing this system when ticks were added to the Market Watch window. This is because we can accurately estimate the time required to create each bar. Therefore the metric code was removed.

Setting the stage for the next test

With all the changes implemented so far, we can move on to the real task: creating a way to model Forex market ticks based solely on the content present in the 1-minute bar files. Trust me, the challenge will be quite significant, but at the same time very interesting and exciting to perform. To make things easier, we will split the C_FileTicks class into 2 classes, but this is just to make the question easier. This split isn't necessary, but since we'll be dealing with some pretty tedious tasks and I don't like classes to have more than 1000 rows, let's split C_FileTicks into two classes.

In this section, we are going to remove the tick modeling part from the C_FileTicks class. The C_Simulation class will be responsible for the conversion of 1-minute bars into ticks so that it can be displayed (and run) without problems. The C_Simulation class will be invisible to the replay system. For the replay service, the data will always come from the real tick file. In fact, they may appear from simulations. Even if you try to access the C_Simulation class from the C_Replay class, it will not be accessible. Thus, everything will work as we expect, since the C_Replay class will only be able to see the C_FileTicks class, which loads the actual ticks present in the file so that the C_Replay class can display them in the MetaTrader 5 terminal.

The new C_FileTicks class declaration now looks like this:

#include "C_FileBars.mqh"
#include "C_Simulation.mqh"
//+------------------------------------------------------------------+
#define macroRemoveSec(A) (A - (A % 60))
//+------------------------------------------------------------------+
class C_FileTicks : private C_Simulation
{

// ... Internal class code 

};

As a result, the C_FileTicks class will privately inherit from the C_Simulation class. This way we will achieve exactly what was described above.

But we need to make small changes to the code of the C_FileTicks class. This is because we inherit the C_Simulation class. But I don't want to send data declared as protected to the C_Simulation class. This is done to ensure that the C_Simulation class remains hidden from the rest of the system. However, we still need to allow the work done by the class to be used by other classes, so we need to add to the following code:

bool BarsToTicks(const string szFileNameCSV)
    {
        C_FileBars  *pFileBars;
        int         iMem = m_Ticks.nTicks,
                    iRet;
        MqlRates    rate[1];
        MqlTick     local[];
        
        pFileBars = new C_FileBars(szFileNameCSV);
        ArrayResize(local, def_MaxSizeArray);
        Print("Converting bars to ticks. Please wait...");
        while ((*pFileBars).ReadBar(rate) && (!_StopFlag))
        {
            ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 3 : def_BarsDiary), def_BarsDiary);
            m_Ticks.Rate[++m_Ticks.nRate] = rate[0];
            iRet = Simulation(rate[0], local);
            for (int c0 = 0; c0 <= iRet; c0++)
            {
                ArrayResize(m_Ticks.Info, (m_Ticks.nTicks + 1), def_MaxSizeArray);
                m_Ticks.Info[m_Ticks.nTicks++] = local[c0];
            }
        }
        ArrayFree(local);
        delete pFileBars;
        m_Ticks.bTickReal = false;
        
        return ((!_StopFlag) && (iMem != m_Ticks.nTicks));
    }

The highlighted lines were part of the code that was executed by the procedure that created the simulation. To keep everything in place, we now need a function that generates the simulation and returns a value. This value is used to tell the C_FileTicks class how many ticks should be stored in an array for later use by the C_Replay class.

We can now focus on the C_Simulation class and build a system that will perform any level of simulation based solely on the data contained in the 1-minute bars file. This will be a subject for consideration in the next topic of this article.

The C_Simulation class

Now that we have divided everything and, in fact, will work with any type of market, we need to determine a small, but very important point. What kind of simulation are we going to create? Do we want the ticks to be the same as in the stock market or forex market? This question seems to create confusion and forces us to use two simulation modes. Think about the following: For you (after all, you are programming the system) this question is relatively simple and purely bureaucratic. But for the user it is something confusing and often they don't really want to know whether the database is being simulated for forex or stock markets. They want the system to work as it should, that is, to simulate data.

But when working with 1-minute bar file data, there is, at first glance, no way to know whether the data in that particular file comes from forex or stock markets. At least, at first glance. But if you look closer at these files and compare the data in them, you may notice a certain pattern. Thanks to this pattern, it is possible to clearly and efficiently define the type of plotting that should be used: Bid or Last. To do this, we need to view the contents of the bars file.

You, as a programmer, should always take on the task of adapting the program to a particular plotting model. The user should not learn the type of modeling present in the 1-minute bars file, because the programmer will solve the problem for the user. If you don't understand what we're talking about, don't worry. I know this may seem like a strange question, but understanding how the database works to solve this particular problem led me to splitting the simulation system code. This allows a better implementation. Before moving on to the C_Simulation class, let's understand how to distinguish a Bid type plotting system from Last.

I think you understand how the system can know whether we are making a replay of data from forex or from the stock market. If you don't understand, I recommend re-reading the previous articles until you really understand how the system can make such a distinction. But if you already understood the part about replay, let's move on to the important question: How can a system know if it will work with data from forex or the stock market when the only information is a file of 1 minute bars? 

In order to find out whether we should use a simulator to create ticks and the information is that used in the forex market or the stock market, we can use the volume! Yes, exactly the volume! That's exactly how it is done. The information that determines whether a bar file belongs to forex (where we use the Bid value as the traded price) or a stock market (where we use Las as the traded price) is precisely the volume.

If you do not understand this, I suggest taking a look at the images below where you can see parts of files with 1-minute barsи.

Figure 02 – Forex asset file

Figure 03 – Stock market asset file

We have no information on whether we are using the Bid or Last as a guide to the actual trade price. In the images above, the only difference is the volume value. This value is highlighted in the images so you can see where the difference is.

Final thoughts on this article

Now let's look at one more point. In the article "Developing a Replay System — Market simulation (Part 11): Birth of the SIMULATOR (I)", when we started developing the simulation system, we used some of the resources present in 1-minute bars to simulate a random walk. As reported in the file, this would be the likely market move within that 1-minute bar. However, at no point in the construction of this mechanism, including in the article "Developing a Replay System — Market simulation (Part 15): Birth of the SIMULATOR (V) - RANDOM WALK", where we constructed a random walk, we considered cases where the volume was zero or the tick volume was very small. But since there are markets and assets that have rather unique meanings, there is a need to cover such cases as well. If you don't do this, the service will hang or crash at some point if you want to run the simulation. Remember that the problem arises in the case of simulation. In the case of replay, everything will work fine, as long as everything is in complete harmony, since there will be no need to work with exotic situations, like what we will do in the future, where we will have to create, or rather simulate, possible market movements.

For now, I’ll let you think about what happened before. The part that deserves careful consideration is the part of creating a simulator for a forex-type market, where we use Bid values for plotting. This is because the number of points that must be explained to understand the system will be very different from what has been observed so far. We will have to reformulate some parts of the simulation calculation. See you in the next article.