Use of custom tick modeling to prevent illusory grail-like backtests of Renko-driven trading robots

Many traders like renko and other "timeless" charts for their clarity and price noise reduction. The charts are used not only for technical analysis but for building trading strategies on top of them. And they require a proper backtesting. However, not many traders realize that Renko backtesting may be (and usually is) highly deceptive due to specificities of the chart and platforms that provide it.

In this post we'll address the pitfalls of renko realization in MetaTrader 5 and how to overcome some of them. But before we begin and for the sake of brevity let us remind you of previous publications on this topic.

The story begins with the introduction of the Renko chart generator - RenkoCharts - it's technical preview (obsolete) first appeared in this blog as well. It builds renko from quotes/rates and do not consider actual price movements inside every bar. If the box size is smaller than a range of specific bar, this may lead to a simplified, unrealistic formation of renko boxes on history. As a result, backtests can run smoother than in reality. Please note that, when online, the generator works by incoming ticks, so the quality of renko is unaffected, but not everyone can afford to collect ticks for months online to build a faithful renko (though it would be unfeasible due to probable interruptions due to bugs, connection losses).

The obvious solution of the problem is to build renko on the history from ticks instead of quotes. Of course, this is possible only if the tick history is available.

Let's take note of the problem and its solution:

A. Avoid building renko from regular timeframe bars, even M1, and use ticks if possible

This approach is implemented in another product - RenkoFromRealTicks.

It provides better quality than the former generator, but does so at the expense of higher resources usage (disk space, bandwidth, CPU load) and execution times.

And guess what? It does not solve all the problems with renko accuracy.

One of them arises from the fact that implementation of renko charts in MetaTrader 5 is based on custom symbols. As far as they are generated locally on your PC and have unique names different to the names of original symbols provided by broker, it's impossible to trade renko symbols online. Ironically, we can trade custom symbols in the tester easily. But this would be theoretical backtests, which can not be mapped to real trading without additional coding and do not guarantee exactly the same (good) results.

To meet renko charts and original symbols your robot should issue orders for original symbols by signals detected on corresponding renkos. There is a dedicated blogpost describing some methods of Adapting EA robots to trading on custom renko charts in MetaTrader 5 and quality metrics of their backtests.

In fact, such robots are multi-symbol ones and should take care of synchronization between at least 2 tickers: custom and original. Basically, you should detect formation of renko boxes, but this is not so trivial as you may think. Nuances exist depending from trading library you use (for example, standard library) and involved tools (such as indicators or rates analysis directly, any of which relies on bars completion). We'll discuss this matter a bit later, in section D.

Now just take note of the task:

B. Ensure cross-symbol synchronization

Some consideration on building more reliable renko-driven tests have been made in another blogpost - Trading on renko charts built from real ticks.

At those stage there were still 3 main problems - their common effect can be described as peeking into the future, which produces unrealistically good trading results:

1. a gap in front of reversal box;
2. every box is M1 bar aligned by 1 minute boundary, whereas its actual formation (either one of open time and end time) happen at some moments inside nearby minutes;
3. fast and prominent price movements may produce boxes ahead of real time, because they are M1 bars (bars can't be placed more tightly to each other than once every 1 minute);

Let us consider the clauses more closely, starting from the 1-st.

Remember that one of possible approaches to adapt EA to work on a custom symbol is to place EA on the chart (renko in our case, which is used for synchronization and technical analysis) but redirect trading orders to original symbol. Normally, all renko signals are taken from closed boxes (because the right-most forming box can end up in any direction, and even its "open" price). And every reversal box is started with the gap, because the tester is capable of simulating regular bars only. In reality, there is no such gap between the close price of the previous box and the open price of the next box. It's just a convention of drawing reversal renko box with the gap.

Execution of trades on 100pt renko chart of EURUSD with artificial ticks or no ticks (left) vs real ticks (right). Note how fake gaps on the left chart minimize losses in comparison to the right chart where there are no gaps.

As you may see, if a trading signal is occured at the moment when reversal box is started, the deal price is shifted from the real price for that moment by the box size (in average). Usually this unfair correction goes in favour of the trader - a losing trade becomes less unprofitable, or profitable trade increases.

If your EA is placed on a chart with original symbol and trades it directly, obtaining signals from boxes and/or indicators on renko symbol, then you eliminate the 1-st problem. But if it's more suitable for you to place EA on renko (for example, EA is built so by a developer and can't be adapted), we'll provide an alternative solution below.

Now let us consider the 2-nd case.

The moment when a price crosses a boundary for next box completion is random. It lies somewhere between two consecutive 1 minute marks. When it happens we create a new box with a timestamp rounded up to nearest whole minute. The real price at this exact minute (without seconds and its fractions) was not the same as the price at the moment when the previous box was closed and the new box was started. The mean of this difference is a half of M1 bar range in average. Again, it means that trades are opened and closed by more beneficial price than in reality, in advance, so to speak. You may think about it as if renko chart is always virtually shifted half-minute ahead of time.

Execution of trades on regular EURUSD chart driven by signal from 100pt renko with artificial ticks or no ticks (left) vs real ticks (right). The signal is formed by upward box completed at 1.08100 on 16:05 M1 bar. The left buy is executed too early (at bar opening, hence with too low price and unexpected bonus), while the right buy is executed at real time 16:05:46 and real price.

The 3-rd problem can be eliminated by choosing sufficiently big box size that time overflows do not happen (not always possible for instruments with large changes in volatility), but overwise it is unfortunately unsolvable by canonical charts, and requires to invent some tricks (lying outside the scope). It doesn't necessarily mean that every one of "run-through" boxes leads to a displaced signal, but some of them can be mishandled.

Execution of the same signals on overflown (looking ahead) boxes when EA is placed on regular M1 chart (left) vs renko (right). Big downside move around 13:30 M1 was mapped into several renko boxes with "future" time, where 2 MAs crossing signal produced at artificial 13:35. When that signal is passed to EA running on regular M1 chart, 13:35 bar stands at earlier moment, as a result the sell is executed in advance with higher (better) price.

One way to probe here is to run EA solely on renko chart (when in the tester). As far as EA takes signals from latest completed box, it's unimportant that the time marks of the boxes may go ahead of real time.

Luckily the 1-st and 2-nd problems mentioned above can be fixed by the following approach:

C. Generate real ticks for renko boxes

Remember that bars (boxes) are just containers for ticks, so an M1 bar doesn't have to start with a tick at 0-th second or end up with a tick at 59-th second. As we create our custom symbol, we can generate ticks for renko box with exact timestamp when the box was open. The first tick will contain exact actual price as well. All other ticks will also contain real prices, but we need to adjust their time to the same timestamp in order to keep them inside the box. If we preserve real time to the renko ticks, the tester and the terminal would form excessive bars (according to they "understanding"), which break proper renko formations.

With such "real" ticks inside renko boxes we can run tests based on real ticks for renko symbols and execute trades with exact prices. No artificial gaps on reversals, no parasitic half-M1 deviations.

So ok, we need the real ticks. The fact that RenkoFromRealTicks builds boxes from real ticks does not mean automatically that there are ticks inside the boxes. Until version 1.4 RenkoFromRealTicks did not generate ticks for the boxes on history.

Now it can.

New parameter TickHistoryMode allows you to choose one of the tick generation modes (in order of increasing accuracy at the expence of resource consumption):

• Single tick per box
• OHLC ticks per box
• Bulk ticks per box

The first mode is almost the same as it was before. It generates a single tick per box in history. The difference is that new versions keep timing with seconds and milliseconds (the old ones used M1 time).

The second mode generates several ticks per box, usually 2 or 3 (depending from whether a box has a wick), which mimics OHLC characteristic prices of a bar (hence the name), with 1 additional warm up tick.

The third mode generates all ticks from original symbol.

You may choose an optimal mode according to your trading algorithm and resources.

When OHLC or bulk ticks modes are enabled, the resulting renko symbol supports testing with the settings "every tick based on real ticks". It's advisable to check tester log to make sure that the real ticks are in effect. For example, if the main symbol, selected in the tester, has tick history, the beginning of the log contains some lines about ticks synchronization (downloading them from the terminal, if it's 1-st run for that symbol, or about synchronized already ticks, otherwise).

Several lines below there should be specific line:

EURUSD_r100,M1 (Your Broker): generating based on real ticks

If you use several renko symbols, all of them should also be mentioned as having real ticks:

...
EURUSD_T_r100: ticks synchronized already [57 bytes]
EURUSD_T_r200: ticks synchronized already [57 bytes]
EURUSD_T_r300: ticks synchronized already [57 bytes]
...
EURUSD_T_r100 : real ticks begin from 2023.01.01 00:00:00
EURUSD_T_r200 : real ticks begin from 2023.01.02 00:00:00
EURUSD_T_r300 : real ticks begin from 2023.01.02 00:00:00
...

Make sure to generate renko history with at least 6-12 months reserve ahead of planned starting date of testing. The dates in the log excerpt above is selected by the tester according to the settings to test from beginning of 2023 year. The renko symbols were generated since the beginning of 2022.

The tester logs this information only for the main tester symbol, for example:

EURUSD_r100: history data begins from 2022.01.02 00:00
EURUSD_r100: ticks data begins from 2022.01.02 00:00

For all additional symbols, loaded from your EA ad-hoc (via CopyRates etc) and added to the tester's market watch, this information is not provided in the log (I don't know why).

If you use other custom symbol generators for comparison or cross-validation, please, keep an eye on further messages in logs. Potentially, even if real ticks are enabled, there can be problems with tick history on some bars. RenkoFromRealTicks does not allow this, but other tools may do.

Here is an example of a custom symbol with 17 problematic bars:

EURUSD_T_r100 : 2023.01.02 23:59 - real ticks absent for 1 minutes out of 16 total minute bars within a day
EURUSD_T_r100 : 2023.01.12 23:59 - real ticks absent for 4 minutes out of 51 total minute bars within a day
EURUSD_T_r100 : 2023.02.03 23:59 - real ticks absent for 3 minutes out of 37 total minute bars within a day
EURUSD_T_r100 : 2023.03.12 23:59 - real ticks absent for 2 minutes out of 8 total minute bars within a day
EURUSD_T_r100 : 2023.03.22 23:59 - real ticks absent for 1 minutes out of 22 total minute bars within a day
EURUSD_T_r100 : 2023.05.05 23:59 - real ticks absent for 1 minutes out of 14 total minute bars within a day
EURUSD_T_r100 : 2023.07.07 23:59 - real ticks absent for 2 minutes out of 18 total minute bars within a day
EURUSD_T_r100 : 2023.09.20 23:59 - real ticks absent for 1 minutes out of 15 total minute bars within a day
EURUSD_T_r100 : 2023.10.08 23:59 - real ticks absent for 1 minutes out of 3 total minute bars within a day
EURUSD_T_r100 : 2023.11.14 23:59 - real ticks absent for 1 minutes out of 18 total minute bars within a day
EURUSD_T_r100 : real ticks begin from 2023.01.01 00:00:00
EURUSD_T_r100 : 2023.01.01 00:00 - 2024.02.01 00:00  real ticks absent for 17 minutes of 3230 total minute bars, every tick generation used

For these bars the tester will generate artificial ticks which are incompatible with renko and may affect accuracy.

One important thing to remember: the tester keeps its own local cache of all symbols used in tests. If you recreate custom symbol with altered properties (including complete removal and building from scratch) the terminal should normally provide new history to the tester, or in other words, the tester should detect the changes and download the history anew. Unfortunately, it doesn't work so at the time of writing, so you need to clean up the tester cache manually if you plan to change custom symbol. The cache is located in the following 2 folders for rates and ticks, correspondingly:

/Tester/bases/<broker>/history/<custom-symbol>/
/Tester/bases/<broker>/ticks/<custom-symbol>/

One may think that when you delete a custom symbol from the terminal, the symbol is automatically deleted from the tester cache as well, but it's not.

D. Implementation

As it's mentioned earlier, a multi-symbol robot needs synchronization between symbols. The signals of renko makes sense only for completed bars, hence we need to check bar completion (or initiation of a new bar, if you wish). As you know, MetaTrader 5 is an asynchronous thing. The incoming ticks are processed by the platform and accumulated as bars, but it may happen with a little delay. In other words, your EA can receive new tick marked by next minute, but corresponding M1 bar is not yet available (is still being constructed). As a result, indices used in expressions like iClose(...,0), iClose(...,1), CopyBuffer(...,1) still returns values from the past (almost outdated, about to be shifted) bars.

Usually this unsychronized state lasts no more than 1 tick, if it happens at all. But it's sufficient to get incorrect signals from old bars and run into losses.

Unfortunately, this aspect is not taken into account in the standard library (if you use MQL5/Include/Expert) and also may not properly handled in other source libraries.

For example, let us look into the method CExpert::Refresh which is called upon new ticks.

bool CExpert::Refresh(void)
{
   MqlDateTime time;
   if(!m_symbol.RefreshRates())
      return(false);
   // check if processing needed
   TimeToStruct(m_symbol.Time(), time);
   if(m_period_flags != WRONG_VALUE && m_period_flags != 0)
      if((m_period_flags & TimeframesFlags(time)) == 0)
         return(false);
   m_last_tick_time = time;
   m_indicators.Refresh();
   return(true);
}

RefreshRates of m_symbol calls SymbolInfoTick internally, and the time from the tick is used as a latch for not processing the bars twice. For that purpose the tick is theoretically mapped in all timeframes (calculated in TimeframesFlags). If the tick advances on one of the timeframes, the code stores m_last_tick_time and refresh indicators. In other cases, the refresh is skipped. But as we already know, the new tick can be not yet propagated to bars, so the refresh (which is actually equivalent of required CopyBuffer's) stores the old data, which is about to change in a fraction of a second. Yet when the rates are updated, and indexing is shifted, the latch does not allow to refresh anymore (because next ticks have times within the theoretical bar, detected at the 1-st tick).

This is why we need to apply some patches to the standard library or do similar stuff in our source codes.

Here is an example, how it could be done in an expert adviser, based on the standard library.

class SyncSymbol
{
  protected:
    const string symbol;
    const ENUM_TIMEFRAMES timeframe;
    const string period;
    datetime lastUnsync;
    long countUnsync;
    
  public:
    SyncSymbol(const string s = NULL, const ENUM_TIMEFRAMES tf = PERIOD_CURRENT):
      symbol(!StringLen(s) ? _Symbol : s), timeframe(tf == PERIOD_CURRENT ? _Period : tf),
      period(StringSubstr(EnumToString(timeframe), StringLen("PERIOD_")))
      {
      }
    
    bool isReady()
    {
      MqlTick t;
      SymbolInfoTick(symbol, t);
      const string now = StringFormat("%s'%03d", TimeToString(t.time, TIME_SECONDS), t.time_msc % 1000);
      if(!SeriesInfoInteger(symbol, timeframe, SERIES_SYNCHRONIZED))
      {
        PrintFormat("%s wait unsync %s %s", now, symbol, period);
        return false;
      }
      
      if(t.time / 60 * 60 != iTime(symbol, timeframe, 0))
      {
        PrintFormat("%s wait unsync with %s on %s %s", now, (string)iTime(symbol, timeframe, 0), symbol, period);
        lastUnsync = t.time / 60 * 60;
        countUnsync++;
        return false;
      }
      
      if(lastUnsync == t.time / 60 * 60 && countUnsync)
      {
        PrintFormat("Synced after %ld ticks on %s %s", countUnsync, symbol, period);
        countUnsync = 0;
      }
      return true;
    }
};
  
void OnTick()
{
  static SyncSymbol symbol;
  
  if(!symbol.isReady()) return;
  
  ExtExpert.OnTick();
}

This demonstrates check-up of current _Symbol only. For multi-symbol EAs it's a good idea to check all symbols in the same manner.

In case of unsynchronization the object will output something like this:

2023.01.03 10:25:53   10:25:53'645 wait unsync with 2023.01.03 09:42:00 on EURUSD_r100 M1
2023.01.03 10:25:53   Synced after 1 ticks on EURUSD_r100 M1

You may find a new revision of the test EA MA2Cross2.mq5 and its signal module attached to the post. Also a couple of important patches for ExpertSignal.mqh and ExpertTrade.mqh are included.

And here are some results achieved with this EA backtested in different modes.

Tester report for trading EURUSD by artificially generated ticks in EURUSD 100pt renko, equivalent of best accuracy provided by former versions of RenkoFromRealTicks (pre-1.4)

Tester report for trading EURUSD by real ticks in EURUSD 100pt renko, with improved accuracy

Tester report for trading directly on EURUSD 100pt renko by real ticks, which demonstrates similar results to previous one and is suitable for validation at smaller resource footprint

You can test EA in different modes and expect online trading performance to be near the lower bound among all results.

F. Final notes

Visualization of the renko in the tester can look strange - boxes may be smaller or larger than predefined size. It can't be fixed - this is how the tester works.

Let us clarify. Renko boxes/bars are properly shaped by the generator in static history and properly re-shaped on-the-fly online. When the ticks are "played" in the visual tester, the generator is out of control, and the bars are formed by ticks only. This is why they look strange.

In this regard, it's important to note that the characteristic price of the renko box is its close price. So, if you use conditions for checking signals, instead of close(i) vs open(i) consider using close(i) vs close(i+1). Open(i) can be somewhere in a range of 0-2 box sizes from close(i) or even equal to close(i), if i-th bar is a single tick bar drawn during big jump of price.

If you're going to use indicators (like MAs) applied to renko, also setup them for close prices exclusively. Though highs and lows should also be reliable.

The similar problems of precise backtesting pertain to other special charts which eliminate a time in a sense that bars are not alined by time scale but formed upon data driven conditions, for example, by volume limit. Also many of such charts have gaps in front of reversal bars, for example Point And Figures.

Further reading:
Mission Impossible: MetaTrader 5 does not support testing and optimization of trading robots based on renko indicators.