Mission Impossible: MetaTrader 5 does not support testing and optimization of trading robots based on renko indicators
This is a continuation of a series of blogposts about trading by signals of renko charts. Before this point we have discussed many aspects of using custom symbols for renko implementation. The latest findings are described in the article Use of custom tick modeling to prevent illusory grail-like backtests of Renko-driven trading robots.
One of the most obvious questions about this series is why do we use custom symbols for Renko implementation rather than something else. In fact, the "something else" is very limited. Drawing Renko on a canvas or using graphical objects are impractical, because it does not allow for backtesting. It seems much more natural to use Renko indicators instead.
Unfortunately, indicators in MetaTrader 5 are designed in such a way that it's impossible to work with Renko indicators in the tester. And here is why.
Renko indicator
To start our research we need an indicator that calculates graphical series which look like Renko boxes. Let's do not invent the wheel and take one of existing indicators, for example, Blue Renko Bars.
As it turned out, this program required to make some bug fixes and improvements, most important of which we'll explain one by one. The final modification is attached to the post.
As the drawing type of the plot is DRAW_CANDLES, the original directive indicator_buffers is incorrect, because this type does not support additional buffer for colors (as opposed to DRAW_COLOR_CANDLES, which utilizes 4+1 buffers).
#property indicator_separate_window #property indicator_buffers 4 // not 5 #property indicator_plots 1
DRAW_CANDLES requires 4 buffers. The buffer array brickColors is useless and has been removed everywhere.
double brickColors[]; The other buffer arrays are initialized at start-up:
int OnCalculate(const int rates_total, // size of input time series const int prev_calculated, // bars handled on a previous call ...) { ... if(prev_calculated == 0) { ArrayInitialize(OpenBuffer, 0); ArrayInitialize(HighBuffer, 0); ArrayInitialize(LowBuffer, 0); ArrayInitialize(CloseBuffer, 0); } ... }
We have introduced new variable lastBar to detect formation of new bar on the host chart. At these moments we need to initialize just added buffer elements (under certain conditions).
datetime lastBar; int OnCalculate(const int rates_total, const int prev_calculated, const datetime &time[],...) { ... if(lastBar != time[0]) { OpenBuffer[0] = 0; HighBuffer[0] = 0; LowBuffer[0] = 0; CloseBuffer[0] = 0; } lastBar = time[0]; ... }
The counter of available renko boxes in renkoBuffer array of MqlRates was not handled correctly for all situations and could produce "out of bound" exception (the indicator would be stopped and unloaded).
int OnCalculate(const int rates_total, // size of input time series const int prev_calculated, // bars handled on a previous call ...) { int size = ArraySize(renkoBuffer); ... // fill array of structs renkoBuffer from M1 rates int first; if(prev_calculated == 0) // checking for the first start of the indicator calculation { ... first = (rates_total > size) ? size : rates_total; // starting number for calculation of all bars } else { // minimum of visible chart bars or RedrawChart can be larger than renkoBuffer size! // moreover, CHART_VISIBLE_BARS is always 0 in the tester in non-visual mode! // (-) MathMax(RedrawChart, ChartGetInteger(0, CHART_VISIBLE_BARS, 0)) first = size; } for(int i = first - 2; i >= 0; i--) { ... // in original indicator i can be larger than renkoBuffer size and produce out of bounds errors HighBuffer[shift + i + 1] = renkoBuffer[i].high; LowBuffer[shift + i + 1] = renkoBuffer[i].low; ... } }
In the function RenkoAdd which adds new box to the renkoBuffer we changed the principle of the operation: instead of heavy ArrayCopy, we wrap the call to ArrayResize with two calls to ArraySetAsSeries.
void RenkoAdd() { int size = ArraySize(renkoBuffer); ArraySetAsSeries(renkoBuffer, false); // (+) ArrayResize(renkoBuffer, size + 1, 10000); ArraySetAsSeries(renkoBuffer, true); // (+) // (-) ArrayCopy(renkoBuffer, renkoBuffer, 1, 0, size); ... }
Also the new element is initilized by empty struct.
void RenkoAdd() { ... const static MqlRates zero = {}; renkoBuffer[0] = zero; }
Attempt N1 (ChartSetSymbolPeriod)
Now let us recall a bit how indicators work and what this means for the renko boxes.
When a new bar is added to the chart, the rates and normal indicators (if applied) are shifted to the left by 1 bar, but the renko boxes should stay still (because new boxes appear by their own "schedule"). On the other hand, when a new box is generated, we need to shift all previous boxes to the left, but rates and other indicators remain at the same position.
It's important to note that any renko indicator must address these problems.
To solve this desynchronization this indicator reserves a variable RedrawChart (which is not even an input parameter) holding a number of bars to redraw. By default it's 1 and is substituted by CHART_VISIBLE_BARS. As a result, renko is correct only on the last CHART_VISIBLE_BARS bars. Moreover, ChartGetInteger(0, CHART_VISIBLE_BARS, 0) returns always 0 bars while testing/optimizing without visual mode! This solution is partial and not universal, potentially leading to miscalculations, if used in automated trading.
Especially, it's flawed in the following aspect. Many trading strategies use a combination of indicators to generate trading signals. For example, all the way through the series of the blogposts we have been using a simple test strategy on 2 MAs crossing on top of renko. To implement it with renko indicator we need to apply MAs to the renko indicator.
And here is the problem: indicators in MetaTrader, calculated once on all bars, are then re-calculated in "short circuit" manner - only on the latest bar. Even if you update CHART_VISIBLE_BARS bars in the renko indicator, the MAs (or other indicators) applied on top of the renko, will update only on the latest bar. As a result, it's impossible to get correct crossing of MAs.
To overcome the problem we've added a new feature to the renko indicator. After new bar creation or after new renko box formation we request the chart to update completely, including all additionally applied indicators. For that puspose the calls to ChartSetSymbolPeriod are added into 2 places: OnCalculate and RenkoAdd.
int OnCalculate(const int rates_total, const int prev_calculated, const datetime &time[],...) { ... if(lastBar != time[0]) { ... ChartSetSymbolPeriod(0, _Symbol, _Period); } lastBar = time[0]; ... } void RenkoAdd() { ... if(lastBar) { ChartSetSymbolPeriod(0, _Symbol, _Period); } }
Now the MAs are properly updated in sync with renko re-positioning.
Renko indicator with applied MA with period 1 (to make sure it works by close prices, CHLO see below)
OHLC -> CHLO
Yet there is another small problem. The renko is represented as 4 buffers with Open, High, Low, and Close prices of the candles. When an additional indicator is applied to another indicator, it uses the very first buffer. Hence our 2 MAs are applied on Open prices, which is not normally a desired effect in the case of renko-based strategy. Instead, the MAs should be applied to the Close prices of the renko boxes. To do so we need to exchange Open and Close buffers in the renko indicator.
The new mode is switched on or off by new parameter SwapOpenClose.
input bool SwapOpenClose = false; // Swap Open & Close int OnInit() { ... if(SwapOpenClose) PlotIndexSetString(0, PLOT_LABEL, "Close;High;Low;Open"); ... } int OnCalculate(...) { ... for(int i = first - 2; i >= 0; i--) { OpenBuffer[i + 1] = SwapOpenClose ? renkoBuffer[i].close : renkoBuffer[i].open; HighBuffer[i + 1] = renkoBuffer[i].high; LowBuffer[i + 1] = renkoBuffer[i].low; CloseBuffer[i + 1] = SwapOpenClose ? renkoBuffer[i].open : renkoBuffer[i].close; ... } }
This looks like a finished renko indicator suitable for trading automation. This is a deception, but we'll discover this a bit later, and will try to add other features to get it to work as expected.
Expert adviser based on 2 MAs crossing on the renko indicator
Currently let's try to adapt our test EA - MA2Cross, originally using renko custom symbols - for working with the renko indicator. The modified version has the name MA2CrossInd.mq5.
Input paramaters are added for underlying indicator:
input int BrickSize = 100; // Brick Size input bool ShowWicks = true; // Show Wicks input bool TotalBars = false; // Use full history input bool SwapOpenClose = true; // Swap Open & Close
The indicator with the given parameters is created in OnInit, and its handle is passed to the signal filter instead of former Signal_2MACross_MAPrice parameter (actually it was Close price all the time).
int OnInit() { ... const int handle = iCustom(_Symbol, _Period, "Blue Renko Bars", BrickSize, ShowWicks, TotalBars, SwapOpenClose); if(handle == INVALID_HANDLE) { Print("Can't create indicator, ", _LastError); return(INIT_FAILED); } ChartIndicatorAdd(0, 1, handle); ... filter0.MAPrice((ENUM_APPLIED_PRICE)handle); // Signal_2MACross_MAPrice ... }
This program could actually trade on an online chart! But it can not be backtested and optimized!
The reason for this is because the function ChartSetSymbolPeriod is not working in the tester. As a result, 2 MAs are not recalculated properly and give incoherent signals.
What can we do?
Attempt N2 (PLOT_SHIFT)
One of ideas was to implement the renko indicator with another principle of re-positioning renko boxes against regular bars. It's based on the property PLOT_SHIFT.
As you know we can shift visual representation of indicator's buffer to the right or to the left, depending from the property: positive values move curves to the right for the specified number of bars, whereas negative values move them to the left. So, when a new regular bar is created, we can apply the shift +1 to our renko boxes to keep them visually on the same place. And when a new renko box is added, we can apply the shift -1 to move old boxes to the left, keeping alignment with the last regular bar.
Because we don't know beforehand the direction in which future price will shift out graph more, we need to make a reserve of empty invisible boxes at the right side. The reserve is provided in corresponding input parameter and used to initialize a variable with current shift.
input int Reserve = 0; int shift; int OnInit() { ... shift = Reserve; PlotIndexSetInteger(0, PLOT_SHIFT, shift); ... }
Then in OnCalculate increase the shift on new bars. Nonzero Reserve is also used as a flag to enable the new mode.
int OnCalculate(...) { ... if(lastBar != time[0]) // new bar added { if(!Reserve) { ChartSetSymbolPeriod(0, _Symbol, _Period); } else { PlotIndexSetInteger(0, PLOT_SHIFT, ++shift); Comment("++", shift); OpenBuffer[0] = 0; HighBuffer[0] = 0; LowBuffer[0] = 0; CloseBuffer[0] = 0; } } ... }
Also decrease the shift on new boxes in RenkoAdd.
void RenkoAdd() { ... if(!Reserve) { ChartSetSymbolPeriod(0, _Symbol, _Period); } else { PlotIndexSetInteger(0, PLOT_SHIFT, --shift); Comment("--", shift); } }
Of course, the shift must be used to adjust indices during writing data into the buffers.
int OnCalculate(...) { ... for(int i = first - 2; i >= 0; i--) { OpenBuffer[shift + i + 1] = SwapOpenClose ? renkoBuffer[i].close : renkoBuffer[i].open; HighBuffer[shift + i + 1] = renkoBuffer[i].high; LowBuffer[shift + i + 1] = renkoBuffer[i].low; CloseBuffer[shift + i + 1] = SwapOpenClose ? renkoBuffer[i].open : renkoBuffer[i].close; if(i == 0) // the forming box is not stored in renkoBuffer yet { OpenBuffer[shift + i] = SwapOpenClose ? close[i] : renkoBuffer[i].close; HighBuffer[shift + i] = upWick ? upWick : MathMax(renkoBuffer[i].close, renkoBuffer[i].open); LowBuffer[shift + i] = downWick ? downWick : MathMin(renkoBuffer[i].close, renkoBuffer[i].open); CloseBuffer[shift + i] = SwapOpenClose ? renkoBuffer[i].close : close[i]; } } }
Unfortunately, despite the fact that the new approach is working perfect visually, the data shift is not detected properly by indicators applied on top of the renko.
This is a known general limitation of MetaTrader platform. The property PLOT_SHIFT can not be detected outside an indicator, and when OnCalculate is called in dependent indicators, they receive and process data unshifted. Let's remind you how the short form of OnCalculate looks like:
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &data[]);
You can see here that indicator receives similar and somewhat related property PLOT_DRAW_BEGIN, which is passed via begin parameter. But there is nothing telling that data array uses shifted indexing. The only way to overcome this is to enter the shift to all indicators as input. But in our case we change the shift dynamically in renko, and hence it's impossible to adjust the shifts in MAs on the fly (unless you re-implement all required indicators yourself and send the shift via chart events, or global variables, or something else - this is too expensive).
Attempt N3 (say "no" to rates_total)
One more approach to try which comes to mind supposes to not update renko boxes at all. Just pile them at the beginning of the tester time and return actual number from OnCalculate, instead of rates_total.
We can use negative Reserve value as a flag to enable this mode. When it's on, buffer indices should be mapped into the range [0..size], which is done by adjusting shift variable.
int OnCalculate(...) { ... if(Reserve < 0) { shift = rates_total - size; if(shift < 0) Print("Renko buffer overflow, will terminate..."); } for(int i = first - 2; i >= 0; i--) { OpenBuffer[shift + i + 1] = SwapOpenClose ? renkoBuffer[i].close : renkoBuffer[i].open; HighBuffer[shift + i + 1] = renkoBuffer[i].high; LowBuffer[shift + i + 1] = renkoBuffer[i].low; CloseBuffer[shift + i + 1] = SwapOpenClose ? renkoBuffer[i].open : renkoBuffer[i].close; ... } ... return(Reserve < 0 ? size + 1 : rates_total); }
In addition, all calls to change PLOT_SHIFT property should be wrapped into appropriate guard conditions:
int OnInit() { ... if(Reserve > 0) { shift = Reserve; PlotIndexSetInteger(0, PLOT_SHIFT, shift); } ... } int OnCalculate(...) { ... if(Reserve > 0) { PlotIndexSetInteger(0, PLOT_SHIFT, ++shift); Comment("++", shift); } ... } void RenkoAdd() { ... else if(Reserve > 0) { PlotIndexSetInteger(0, PLOT_SHIFT, --shift); Comment("--", shift); } }
The indicator is calculated and displayed properly in this mode while running in the visual tester (remember, you need to scroll to the beginning of the chart to see the boxes). But this is another deception.
If it would work as expected, we'd need to change the signal generation module Signal2MACross.mqh slightly by adding the following method (see attached Signal2MACrossDEMA.mqh):
class Signal2MACross : public CExpertSignal { ... int StartIndex(void) override { const int base = iBars(_Symbol, PERIOD_CURRENT) - ::BarsCalculated(m_type); return((m_every_tick ? base : base + 1)); // it was (m_every_tick ? 0 : 1) } }
Here m_type is a type of price on which MA indicators are applied, and we can assign a handle of the renko indicator to it, as it was mentioned before:
... const int handle = iCustom(_Symbol, _Period, "Blue Renko Bars", BrickSize, ShowWicks, TotalBars, SwapOpenClose); ... filter0.MAPrice((ENUM_APPLIED_PRICE)handle); ...
This way the renko close price from our EA will be used for MA calculations.
Unfortunately, all of this is not working because the value returned from BarsCalculated is always rates_total, not actual value returned from OnCalculate.
We're attaching a slightly modified example of DEMA indicator (DEMAtest.mq5, to be placed in MQL5/Indicators/Examples/), which allows you to trace and output actual OnCalculate's parameters received by the indicator, when it's applied on the renko indicator with reduced number of calculated bars (boxes). This indicator is also used in the Signal2MACrossDEMA.
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &price[]) { const int limit = (_AppliedTo >= 10) ? BarsCalculated(_AppliedTo) : rates_total; // not working as expected! ... #ifdef _DEBUG Print(rates_total, " ", prev_calculated, " ", limit, " ", begin); #endif ... }
You can make sure that the number of available bars is always reported as rates_total, and as a result - the signal module above will read data at incorrect indices.
I consider this a bug of the platform, because it's passing correct values via begin parameter, but not via rates_total.
Bottom line
At the time of writing, it's impossible in MetaTrader 5 to run a backtest or optimization of EA which is based on signals of a renko indicator. There are some workarounds available.
You can use custom symbols with renko, as it's described in my previous blogposts.
You can calculate Renko virtually inside EA. This may become a hard routine task if you need to apply different technical indicators to Renko, because you need to re-implement them from scratch for your virtual structures.
Or you can use only a limited subset of signals relying to renko boxes (without additional indicators), for example, checking Close[i] against each other. Blue Renko Bars indicator is ready for this scenario.
PS
After first publication, the indicator Blue Renko Bars has been updated (04.03.2024):
- critical bug fix in the function Renko;
- drawing loop is optimized: filling buffers is limited to latest changed boxes only;
- automatic refresh is requested via timer in case of missing data;
Further reading:
Tweaking renko indicators for testing and optimizing robots dependent on them
