Developing a Replay System (Part 62): Playing the service (III)

Introduction

In the previous article Developing a Replay System (Part 61): Playing the service (II), I explained an issue we are currently facing when using the simulation mode in our system. This issue does not necessarily stem from a catastrophic failure in the application we are developing. Rather, it is due to the system's overall response speed. The response time was not enough for the application to properly process all the incoming data. As a result, we must make certain adjustments. Even if our service does not perfectly align with an ideal scenario, we recognize that such an ideal scenario rarely exists in practice.

The best solution I could envision was to adjust the maximum limits applicable in the simulation. However, throughout this article, I will carefully explore the effect of these changes and explain why I chose this particular approach. Beyond this, there is another factor that is directly related to real or externally simulated data outside the application being developed. As unusual as it may seem, in some cases, especially in relation to futures contracts, we may encounter an exceptionally high number of ticks or trades within a single one-minute bar. When this occurs, even when connected to the trading server, we experience issues related to the speed at which the MetaTrader 5 platform processes and displays price movements. If you have never encountered this issue before, you might assume it is due to limitations in the hardware running MetaTrader 5 or an operating system malfunction. However, I regret to inform you that such assumptions are entirely unfounded - misconceptions spread by those who lack a proper understanding of computing.

Given that we face these challenges even when connected to a real trading server, where the platform struggles to process the vast influx of incoming data, the situation becomes even more problematic when replaying such data. It would be a complete disaster, as timing accuracy would deteriorate significantly. Therefore, we will also establish a limit for real or externally simulated data to prevent the platform's data processing limitations from becoming evident or causing further issues. Now, let's examine how the new code will be structured.

A New Concept. A New System

Perhaps the title of this section is not entirely self-explanatory - it does not fully convey what we are about to implement. However, we will begin by analyzing the changes made to the class responsible for generating and modeling any simulation within our system. Before reviewing the simulation class code, we must first examine the definition file, as a new line has been added, which will be referenced at various points in the code.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. #define def_VERSION_DEBUG
05. //+------------------------------------------------------------------+
06. #ifdef def_VERSION_DEBUG
07.    #define macro_DEBUG_MODE(A) \
08.                Print(__FILE__, " ", __LINE__, " ", __FUNCTION__ + " " + #A + " = " + (string)(A));
09. #else
10.    #define macro_DEBUG_MODE(A)
11. #endif
12. //+------------------------------------------------------------------+
13. #define def_SymbolReplay      "RePlay"
14. #define def_MaxPosSlider       400
15. #define def_MaxTicksVolume     2000
16. //+------------------------------------------------------------------+
17. union uCast_Double
18. {
19.    double    dValue;
20.    long      _long;                                 // 1 Information
21.    datetime _datetime;                              // 1 Information
22.    uint     _32b[sizeof(double) / sizeof(uint)];    // 2 Informations
23.    ushort   _16b[sizeof(double) / sizeof(ushort)];  // 4 Informations
24.    uchar    _8b [sizeof(double) / sizeof(uchar)];   // 8 Informations
25. };
26. //+------------------------------------------------------------------+
27. enum EnumEvents    {
28.          evHideMouse,               //Hide mouse price line
29.          evShowMouse,               //Show mouse price line
30.          evHideBarTime,             //Hide bar time
31.          evShowBarTime,             //Show bar time
32.          evHideDailyVar,            //Hide daily variation
33.          evShowDailyVar,            //Show daily variation
34.          evHidePriceVar,            //Hide instantaneous variation
35.          evShowPriceVar,            //Show instantaneous variation
36.          evSetServerTime,           //Replay/simulation system timer
37.          evCtrlReplayInit,          //Initialize replay control
38.                   };
39. //+------------------------------------------------------------------+

Defines.mqh file source code

It is line 15 that has been added to the code. I won't explain now where this value comes from, we'll talk about it when we use it in the simulation code. This will be the first place where it will be used. Now that we've covered this change, we can move on to the source code of the class responsible for simulating ticks. The code with the changes made is shown below:

001. //+------------------------------------------------------------------+
002. #property copyright "Daniel Jose"
003. //+------------------------------------------------------------------+
004. #include "..\..\Defines.mqh"
005. //+------------------------------------------------------------------+
006. class C_Simulation
007. {
008.    private   :
009. //+------------------------------------------------------------------+
010.       int      m_NDigits;
011.       bool     m_IsPriceBID;
012.       double   m_TickSize;
013.       struct st00
014.       {
015.          bool  bHigh, bLow;
016.          int   iMax;
017.       }m_Marks;
018. //+------------------------------------------------------------------+
019. template < typename T >
020. inline T RandomLimit(const T Limit01, const T Limit02)
021.          {
022.             T a = (Limit01 > Limit02 ? Limit01 - Limit02 : Limit02 - Limit01);
023.             return (Limit01 >= Limit02 ? Limit02 : Limit01) + ((T)(((rand() & 32767) / 32737.0) * a));
024.          }
025. //+------------------------------------------------------------------+
026. inline void Simulation_Time(const MqlRates &rate, MqlTick &tick[])
027.          {
028.             for (int c0 = 0, iPos, v0 = (int)(60000 / m_Marks.iMax), v1 = 0, v2 = v0; c0 <= m_Marks.iMax; c0++, v1 = v2, v2 += v0)
029.             {
030.                iPos = RandomLimit(v1, v2);
031.                tick[c0].time = rate.time + (iPos / 1000);
032.                tick[c0].time_msc = iPos % 1000;
033.             }
034.          }
035. //+------------------------------------------------------------------+
036. inline void CorretTime(MqlTick &tick[])
037.          {
038.             for (int c0 = 0; c0 <= m_Marks.iMax; c0++)
039.                tick[c0].time_msc += (tick[c0].time * 1000);
040.          }
041. //+------------------------------------------------------------------+
042. inline int Unique(const double price, const MqlTick &tick[])
043.          {
044.             int iPos = 1;
045.             
046.             do
047.             {
048.                iPos = (m_Marks.iMax > 20 ? RandomLimit(1, m_Marks.iMax - 1) : iPos + 1);
049.             }while ((m_IsPriceBID ? tick[iPos].bid : tick[iPos].last) == price);
050.             
051.             return iPos;
052.          }
053. //+------------------------------------------------------------------+
054. inline void MountPrice(const int iPos, const double price, const int spread, MqlTick &tick[])
055.          {
056.             if (m_IsPriceBID)
057.             {
058.                tick[iPos].bid = NormalizeDouble(price, m_NDigits);
059.                tick[iPos].ask = NormalizeDouble(price + (m_TickSize * spread), m_NDigits);
060.             }else
061.                tick[iPos].last = NormalizeDouble(price, m_NDigits);
062.          }
063. //+------------------------------------------------------------------+
064. inline void Random_Price(const MqlRates &rate, MqlTick &tick[])
065.          {
066.             for (int c0 = 1; c0 < m_Marks.iMax; c0++)
067.             {
068.                MountPrice(c0, NormalizeDouble(RandomLimit(rate.high, rate.low), m_NDigits), (rate.spread + RandomLimit((int)(rate.spread | (m_Marks.iMax & 0xF)), 0)), tick);
069.                m_Marks.bHigh = (rate.high == (m_IsPriceBID ? tick[c0].bid : tick[c0].last)) || m_Marks.bHigh;
070.                m_Marks.bLow = (rate.low == (m_IsPriceBID ? tick[c0].bid : tick[c0].last)) || m_Marks.bLow;
071.             }
072.          }
073. //+------------------------------------------------------------------+
074. inline void DistributeVolumeReal(const MqlRates &rate, MqlTick &tick[])
075.          {
076.             for (int c0 = 0; c0 <= m_Marks.iMax; c0++)
077.             {
078.                tick[c0].volume_real = 1.0;
079.                tick[c0].volume = 1;
080.             }
081.             if ((m_Marks.iMax + 1) < rate.tick_volume) for (int c0 = (int)(rate.tick_volume - m_Marks.iMax); c0 > 0; c0--)
082.                tick[RandomLimit(0, m_Marks.iMax - 1)].volume += 1;
083.             for (int c0 = (int)(rate.real_volume - m_Marks.iMax); c0 > 0; c0--)
084.                tick[RandomLimit(0, m_Marks.iMax)].volume_real += 1.0;
085.          }
086. //+------------------------------------------------------------------+
087. inline int RandomWalk(int In, int Out, const double Open, const double Close, double High, double Low, const int Spread, MqlTick &tick[], int iMode, int iDesloc)
088.          {
089.             double vStep, vNext, price, vH = High, vL = Low;
090.             char i0 = 0;
091.             
092.             vNext = vStep = (Out - In) / ((High - Low) / m_TickSize);
093.             for (int c0 = In, c1 = 0, c2 = 0; c0 <= Out; c0++, c1++)
094.             {
095.                price = (m_IsPriceBID ? tick[c0 - 1].bid : tick[c0 - 1].last) + (m_TickSize * ((rand() & 1) == 1 ? -iDesloc : iDesloc));
096.                price = (price > vH ? vH : (price < vL ? vL : price));
097.                MountPrice(c0, price, (Spread + RandomLimit((int)(Spread | (m_Marks.iMax & 0xF)), 0)), tick);
098.                switch (iMode)
099.                {
100.                   case 1:
101.                      i0 |= (price == High ? 0x01 : 0);
102.                      i0 |= (price == Low ? 0x02 : 0);
103.                      vH = (i0 == 3 ? High : vH);
104.                      vL = (i0 ==3 ? Low : vL);
105.                      break;
106.                   case 0:
107.                      if (price == Close) return c0;
108.                   default:
109.                      break;
110.                }
111.                if (((int)floor(vNext)) >= c1) continue; else if ((++c2) <= 3) continue;
112.                vNext += vStep;
113.                vL = (iMode != 2 ? (Close > vL ? (i0 == 3 ? vL : vL + m_TickSize) : vL) : (((c2 & 1) == 1) ? (Close > vL ? vL + m_TickSize : vL) : (Close < vH ? vL : vL + m_TickSize)));
114.                vH = (iMode != 2 ? (Close > vL ? vH : (i0 == 3 ? vH : vH - m_TickSize)) : (((c2 & 1) == 1) ? (Close > vL ? vH : vH - m_TickSize) : (Close < vH ? vH - m_TickSize : vH)));
115.             }
116.             
117.             return Out;
118.          }
119. //+------------------------------------------------------------------+
120.    public   :
121. //+------------------------------------------------------------------+
122.       C_Simulation(const int nDigits)
123.          {
124.             m_NDigits       = nDigits;
125.             m_IsPriceBID    = (SymbolInfoInteger(def_SymbolReplay, SYMBOL_CHART_MODE) == SYMBOL_CHART_MODE_BID);
126.             m_TickSize      = SymbolInfoDouble(def_SymbolReplay, SYMBOL_TRADE_TICK_SIZE);
127.          }
128. //+------------------------------------------------------------------+
129. inline int Simulation(const MqlRates &rate, MqlTick &tick[], const int MaxTickVolume = def_MaxTicksVolume)
130.          {
131.             int    i0, i1, i2;
132.             bool   b0;
133.             
134.             m_Marks.iMax   = (MaxTickVolume <= 0 ? 1 : (MaxTickVolume >= def_MaxTicksVolume ? def_MaxTicksVolume : MaxTickVolume));
135.             m_Marks.iMax   = ((int)rate.tick_volume > m_Marks.iMax ? m_Marks.iMax : (int)rate.tick_volume - 1);
136.             m_Marks.bHigh  = (rate.open == rate.high) || (rate.close == rate.high);
137.             m_Marks.bLow   = (rate.open == rate.low) || (rate.close == rate.low);
138.             Simulation_Time(rate, tick);
139.             MountPrice(0, rate.open, rate.spread, tick);
140.             if (m_Marks.iMax > 10)
141.             {
142.                i0 = (int)(MathMin(m_Marks.iMax / 3.0, m_Marks.iMax * 0.2));
143.                i1 = m_Marks.iMax - i0;
144.                i2 = (int)(((rate.high - rate.low) / m_TickSize) / i0);
145.                i2 = (i2 == 0 ? 1 : i2);
146.                b0 = (m_Marks.iMax >= 1000 ? ((rand() & 1) == 1) : (rate.high - rate.open) < (rate.open - rate.low));
147.                i0 = RandomWalk(1, i0, rate.open, (b0 ? rate.high : rate.low), rate.high, rate.low, rate.spread, tick, 0, i2);
148.                RandomWalk(i0, i1, (m_IsPriceBID ? tick[i0].bid : tick[i0].last), (b0 ? rate.low : rate.high), rate.high, rate.low, rate.spread, tick, 1, i2);
149.                RandomWalk(i1, m_Marks.iMax, (m_IsPriceBID ? tick[i1].bid : tick[i1].last), rate.close, rate.high, rate.low, rate.spread, tick, 2, i2);
150.                m_Marks.bHigh = m_Marks.bLow = true;
151. 
152.             }else Random_Price(rate, tick);
153.             if (!m_IsPriceBID) DistributeVolumeReal(rate, tick);
154.             if (!m_Marks.bLow) MountPrice(Unique(rate.high, tick), rate.low, rate.spread, tick);
155.             if (!m_Marks.bHigh) MountPrice(Unique(rate.low, tick), rate.high, rate.spread, tick);
156.             MountPrice(m_Marks.iMax, rate.close, rate.spread, tick);
157.             CorretTime(tick);
158. 
159.             return m_Marks.iMax;
160.          }
161. //+------------------------------------------------------------------+
162. };
163. //+------------------------------------------------------------------+

Source code of C_Simulation.mqh

At this point, you may not immediately notice any changes, especially since it has been quite some time since we last modified this class. The last time we worked on it was in the article on Random Walk. However, because we need to ensure the system maintains some level of consistency in terms of timing, even with some degradation in the number of generated ticks, a slight modification to this code was necessary.

To help you understand what has changed, I will highlight one key aspect. Although there are several minor adjustments throughout the code that do not need specific attention, as they were made solely to align the process with a new methodology. The most significant change, in this case, is an addition in line 16. This variable did not originally exist in the class code. However, before examining its broader implications for the entire class, let's first see where it is initialized. You might assume that its initialization occurs in the class constructor. But that is not the case. Instead, it takes place between lines 134 and 135. Pay close attention to what happens in these two lines. This will be crucial if you plan to modify the system. In line 129, we declare the function responsible for generating the tick simulation. However, we now introduce an additional parameter that specifies the maximum number of ticks to be simulated. Remember the line added to the Defines.mqh file? One of the places where that definition is utilized is precisely here. Let's analyze what is happening. This way, if you decide to modify the code, you will understand how your adjustments will influence its behavior. When calling the function to execute the tick simulation, you must also provide a value indicating the maximum number of ticks to be simulated. This value is not mandatory, as it already has a default value. However, if the provided value is less than or equal to zero, the system will assume a minimum of one tick. This is not an arbitrary decision. Rather, it is because, in Forex trading, the smallest possible tick volume is precisely one. If you specify a value exceeding the predefined limit, the class will ignore your input and use the system-defined maximum value instead. This limit is set in the Defines.mqh header file and determines the maximum number of ticks that can be simulated inside the system. Any value within these two extremes will be used as the maximum tick count for the simulation. Thus, you can adjust it within this range.

Now one important detail: this specific maximum value was not selected at random. If we divide a one-minute bar by this limit (set to 2000 ticks) each tick is spaced approximately 30 milliseconds apart. This interval is considered optimal, ensuring smooth and consistent motion throughout the plotting process.

While you can specify higher values, be aware that this will not increase the actual number of ticks simulated. Instead, it will only raise the upper limit that can be simulated. This explanation applies to line 134, but the actual maximum tick count is determined in line 135. At this point, when line 135 establishes the final tick count, it verifies the value generated in line 134 against the data present in the bar. If the value from line 134 is lower than the tick count in the bar, it is used. If the bar's tick count is lower, the provided input is disregarded, and the bar's tick count is used instead.

As mentioned earlier, these modifications required a complete review of all tests related to the maximum tick count. Consequently, all functions and procedures within this class have undergone minor changes. Since these changes are straightforward, I will not delve into their detailed descriptions. If you have any questions, you can refer to the article on random walk. Check the link: "Developing a Replay System — Market simulation (Part 15): Birth of the SIMULATOR (V) - RANDOM WALK".

Very well. So, the first problem is solved. Now, every time we run a simulation, we can adjust the maximum number of ticks required to create a bar. However, this is not enough for our purposes. Remember: we only control the tick simulation process, but we do not offer the user the ability to change the settings to set a smaller number of simulated ticks. So we have two problems to solve. Both of them require some changes to the system.

So let's take action. The next question to be addressed is the maximum number of real or externally simulated ticks that can be present. To consider this, let's move on to the next topic.

Adjusting for Real Market Data

Despite the title suggesting that only real market data will be used, this is not entirely true. You can externally simulate market movements, save them in a file, and later use this file to provide tick data instead of bars. This is a perfectly valid and feasible approach. However, the core issue remains the same as in the previous section: we need to set a limit on the maximum number of ticks, even when dealing with real market data.

Unlike the previous case, this problem is significantly more complex due to the nature of the data being processed. If the replay/simulation service were designed for a single market type, it would be relatively easy to resolve. In this case, we could perform an analysis while loading ticks and check if the sum within the time window exceeded a given value. If it did, we would force the system to discard exceeding ticks within this window, which is equivalent to a minute bar. We then would have the simulator generate movement based on a RANDOM WALK. This way it would be easy to limit the number of ticks in a window. However, since we don't know whether we're dealing with BID or LAST values when loading ticks, we have a tricky problem to solve. And the difficulty lies precisely in the fact that we do not know which charting system is used: BID or LAST.

One of the first questions to consider is what is the best solution to the problem? Do we need to allow the user to specify the charting type and risk getting it wrong, or modify the code to handle this situation? Although the user-specified charting type solution is much easier to implement, it has one drawback: the user may incorrectly specify it. Let's be honest: how many users do really understand that there are two types of chart representations and that one of them corresponds to a certain market model, and the other to a completely different model? The majority of users do not know that MetaTrader 5 supports three types of servers, let alone that they need to configure the correct plotting method for their specific asset.

Due to this potential pitfall, our implementation will require additional effort. However, we now have a clear starting point: the LoadTicks function within the C_FileTicks class. Let's now look at the original function and think about what we should implement. This function is shown below:

01.       datetime LoadTicks(const string szFileNameCSV, const bool ToReplay = true)
02.          {
03.             int      MemNRates,
04.                      MemNTicks;
05.             datetime dtRet = TimeCurrent();
06.             MqlRates RatesLocal[],
07.                      rate;
08.             bool     bNew;
09.             
10.             MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
11.             MemNTicks = m_Ticks.nTicks;
12.             if (!Open(szFileNameCSV)) return 0;
13.             if (!ReadAllsTicks()) return 0;         
14.             rate.time = 0;
15.             for (int c0 = MemNTicks; c0 < m_Ticks.nTicks; c0++)
16.             {
17.                if (!BuildBar1Min(c0, rate, bNew)) continue;
18.                if (bNew) ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
19.                m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
20.             }
21.             if (!ToReplay)
22.             {
23.                ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
24.                ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
25.                CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
26.                dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
27.                m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
28.                m_Ticks.nTicks = MemNTicks;
29.                ArrayFree(RatesLocal);
30.             }else SetSymbolInfos();
31.             m_Ticks.bTickReal = true;
32.                            
33.             return dtRet;
34.          }; 

C_FilesTicks.mqh source code fragment

Don't worry about the line numbers. Here, they serve only as a reference for the explanation and do not represent the actual lines within the file C_FileTicks.mqh, as it has already undergone modifications that will be reviewed later. Now, let's understand what happens during the reading of real ticks.

When the function begins, we have, in lines 10 and 11, two points where we temporarily store the current values of the loaded tick positions and the bars that represent these ticks. If the caller specifies that ticks will not be used as the basis for replay but instead as pre-existing bars, these values will be replaced in lines 27 and 28, respectively. This ensures that the system remains intact, awaiting the replay ticks.

In line 12, we attempt to open the file containing the data, and in line 13, we read all (absolutely all) ticks present in the file. If the reading is successful, we will have all ticks loaded. However, in some instances, the number of ticks per unit of time may exceed the system-defined maximum. At this point, there is nothing we can do about it. The reason is that we do not yet know which type of charting will be used. But once all the ticks have been fully read, we will determine the charting type and can start working on a solution.

Now comes the interesting part. We will analyze a one-minute time interval. Here's what we do. In line 15, we enter a loop that aims to construct one-minute bars, allowing the system to access them when needed. This is the point where we will intervene. In line 17, we call the function responsible for constructing the bars. This function will account for the volume of ticks used to generate movement within the bar. Now, pay close attention: when the condition in line 18 evaluates to true, the bar data will be displayed in the same way as if it had been read from a bar file. This is precisely the data we need to pass to the simulation class. You can check this by revisiting the previous topic and looking at line 129.

Do you see the implementation idea we need? We must take an action when it is determined that the number of ticks exceeds the internally defined value in the program, at least in this initial stage of implementation. We will make changes related to this later.

This part is straightforward: check the tick count and, if necessary, instruct the simulation class to perform a RANDOM WALK to ensure the correct number of ticks for movement. Now comes the complicated part. The tick simulation class will generate the ticks, but we need the simplest possible method to modify the loaded ticks. However, we will only remove ticks within the time window where the tick count exceeds the internally defined or application-specified limit. In addition to this issue, we have another, albeit simpler, problem. The simulation should only take place if the data is intended for replay usage. Fortunately, this is easy to resolve, as the caller informs us whether the data will be used for replay. All we need to do is check the ToReplay value. Piece of cake! Now, let's try to solve the difficult part: efficiently overriding unnecessary ticks. To achieve this, we will modify the function shown in the previous fragment and replace it with another.

The first attempt at making this work (developing new features is always a process of trial and error) is shown in the following fragment:

01. //+------------------------------------------------------------------+
02.       datetime LoadTicks(const string szFileNameCSV, const bool ToReplay, const int MaxTickVolume)
03.          {
04.             int      MemNRates,
05.                      MemNTicks,
06.                      nDigits,
07.                      nShift;
08.             datetime dtRet = TimeCurrent();
09.             MqlRates RatesLocal[],
10.                      rate;
11.             MqlTick  TicksLocal[];
12.             bool     bNew;
13.             
14.             MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
15.             nShift = MemNTicks = m_Ticks.nTicks;
16.             if (!Open(szFileNameCSV)) return 0;
17.             if (!ReadAllsTicks()) return 0;         
18.             rate.time = 0;
19.             nDigits = SetSymbolInfos(); 
20.             ArrayResize(TicksLocal, def_MaxSizeArray);
21.             m_Ticks.bTickReal = true;
22.             for (int c0 = MemNTicks, c1, MemShift = nShift; c0 < m_Ticks.nTicks; c0++, nShift++)
23.             {
24.                if (!BuildBar1Min(c0, rate, bNew)) continue;
25.                if (bNew)
26.                {
27.                   if ((m_Ticks.nRate >= 0) && (ToReplay)) if (m_Ticks.Rate[m_Ticks.nRate].tick_volume > MaxTickVolume)
28.                   {
29.                      nShift = MemShift;
30.                      C_Simulation *pSimulator = new C_Simulation(nDigits);
31.                      if ((c1 = (*pSimulator).Simulation(m_Ticks.Rate[m_Ticks.nRate], TicksLocal, MaxTickVolume)) < 0) return 0;
32.                      ArrayCopy(m_Ticks.Info, TicksLocal, nShift, 0, c1);
33.                      nShift += c1;
34.                      delete pSimulator;
35.                   }
36.                   MemShift = nShift;
37.                   ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
38.                };
39.                m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
40.             }
41.             ArrayFree(TicksLocal);
42.             if (!ToReplay)
43.             {
44.                ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
45.                ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
46.                CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
47.                dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
48.                m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
49.                m_Ticks.nTicks = MemNTicks;
50.                ArrayFree(RatesLocal);
51.             }else m_Ticks.nTicks = nShift;
52.                            
53.             return dtRet;
54.          };
55. //+------------------------------------------------------------------+

Source code fragment of the C_FileTicks.mqh class

You may notice that several things have been added here. Additionally, some modifications have been made to the execution sequence of certain commands. However, even though this fragment may seem like the solution, it still contains a small flaw. But before we get to that, let's analyze how this code fragment in the C_FileTicks class effectively prevents the number of real ticks from exceeding the internally defined system limit.

To accomplish this, lines 6, 7, and 11 were added. These lines introduce new variables that we will actually need. In line 15, we initialize the first of these new variables to track the current count of loaded ticks. Lines 18 and 19 were also added, though they only serve to initialize some values. The added line 20 is necessary to allocate memory for storing the ticks that will be simulated. This allocated memory is only released in line 41. Yet, there is a small flaw here, which is related to line 31. We will correct in the final version of the code. But hold on, we'll get there.

In line 21 we find something very important. In this fragment, the reason may not be apparent. However, if the content of line 21 were placed in its original position in the code, it would fail to perform its intended function. You can compare this by looking at the original fragment above and identifying where line 21 from this fragment originally appears. However, when line 24 executes, the BuildBar1Min function will not be able to establish the necessary criteria. That's why the execution order must follow the sequence shown in this new fragment.

Regardless, the implementation of what was described as necessary is contained between lines 27 and 35. This section specifically handles the simulation of ticks to be used when the tick count in a one-minute bar exceeds a predefined value. This value is internally set within the application.

Now pay attention to the following. Line 27 contains two checks. The first determines whether the data will be used in replay mode or as pre-existing bars. If the data will be used as pre-existing bars, simulation will not be necessary. The second check determines whether the number of ticks exceeds the predefined limit. However, since this second check may refer to a negative index, the first check is designed to prevent such an issue. As a result, when performing the second check, we ensure that it refers to the current index, which has just had its tick volume recorded.

But wait a minute. The condition in line 25 ensures that these checks only take place when a new bar is detected. So how can we be analyzing the recorded bar data when we are already in a new bar? If you thought about this, congratulations! That means you are truly understanding the code. However, you may have overlooked one small but crucial detail. Until line 39 executes, we are still looking at the bar that has just closed. The actual transition to a new bar only occurs in line 39. Do you see now why the execution sequence must be carefully structured?

Now, let's return to the simulation process. With the exception of line 51, all other parts continue functioning as described in previous articles in this series. Line 51 depends on what happens during the tick simulation phase. So pay close attention, as there is a flaw here, though it is not significantly detrimental to testing this function.

In line 29, we set the displacement pointer to the starting position of the ticks in the bar that will be replaced. Then, in line 30, we initialize the simulation system. In line 31, we start the actual simulation. Note that if the simulation fails, line 41 will not execute, meaning the allocated memory will not be released. While this is a minor issue for testing purposes, it will be corrected later. Once the tick simulation is successful, we make a function call in line 32. Now, take note: this line 32 could be replaced by a for loop, but since the MQL5 library routine is likely optimized for fast data transfer, it is preferable to use it instead of a loop here. In line 33, we update the displacement value to point to a new position immediately after the data transfer. Finally, in line 34, we destroy the simulator.

To ensure that the position memory is updated so that in line 29 we correctly point to the intended location, we use line 36 to perform this update. However, this entire procedure contains a few minor flaws, which will be addressed in the next article. Additionally, we need to correct another issue, one that remains almost hidden but is present in the simulation phase.

Conclusion

Despite making significant progress in limiting the number of ticks per one-minute bar, these advances have introduced new problems and revealed others. Given that this article is already quite dense and requires careful study to fully grasp what is happening, I do not want to introduce even more complexity. However, if you want to challenge yourself by identifying the remaining flaws that emerged during the implementation of this code, here's a hint: one of the flaws involves the minimum number of ticks that should be simulated. This is an interesting issue to fix. This flaw only became apparent because we now want to simulate ticks when their count exceeds a given value. Take a moment to think about it, and you should be able to understand how this happens. The other flaw occurs when copying simulated values into the tick array. When this copy takes place, the replay bar generation system is compromised. Sometimes it may generate illogical or erratic patterns. In addition, at certain times the ticks simply disappear, which prevents the replay system from working accurately and correctly.

If you want to try fixing these issues before reading the next article - great! This will be excellent training to help you understand how to troubleshoot, develop, and refine your own solutions. Regardless, in the next article, I will show how to fix these flaws. This will be very interesting. So see you in the next article.

Video demonstrating a system flaw

Soon, I will show how to fix this issue, though not immediately, as it is not critical. This problem is only related to loading or unloading created objects. This presents a great opportunity. If you really want to test your programming skills, try fixing the issue before I show you how. You don't need to show me your solution, but give it a try before reading my solution. This will help you evaluate the level of learning you are at now.