AO Core
To ensure self-optimization of the advisor for implementing any required capabilities and functionalities, the scheme presented in Figure 1 is employed.
On the "History" timeline, the advisor is positioned at the "time now" point where the optimization decision is made. The "EA" advisor invokes the "Manager function" which manages the optimization process, with the advisor passing optimization settings, "optimization parameters" to this function.
In turn, the manager requests a set of parameters from the optimization algorithm, "optimization ALGO" or "AO" which will be referred to as the "set" from now on. Subsequently, the manager passes the set to the virtual trading strategy, "EA Virt" which is a complete analogue of the real strategy, executing trading operations, "EA".
"EA Virt" engages in virtual trading from the "past" point in history to the "time now" point. The manager initiates the execution of "EA Virt" as many times as specified by the population size in the "optimization parameters". "EA Virt" then returns the historical run results in the form of "ff result".
"ff result" represents the fitness function result or optimization criterion, which can be anything at the user's discretion. This could be, for example, balance, profit factor, mathematical expectation, or a complex criterion, integral, or aggregate differential, measured at various points in time on the "History" timeline. Thus, the fitness function result, or "ff result" is what the user deems a crucial indicator of trading strategy quality.
Subsequently, the "ff result" an evaluation of a specific set, is passed by the manager to the optimization algorithm.
Upon meeting the stop condition, the manager transfers the best set to the trading advisor "EA" after which the advisor continues its operations (trading) with updated parameters from the "time now" point to the re-optimization point "reoptimiz" where a re-optimization is conducted to a specified depth in history.
The re-optimization point can be chosen based on various considerations; it could be a fixed number of historical bars as in the example provided below, or a specific condition such as a decrease in trading metrics to a critical level.
Figure 1.
According to the operation scheme of the optimization algorithm "optimization ALGO, "it can be viewed as a "black box" that functions autonomously (indeed, to it, everything external is also a "black box"), regardless of the specific trading strategy, manager, and virtual strategy. The manager requests a set from the optimization algorithm and sends back an evaluation of this set, which the optimization algorithm utilizes to determine the next set. This cycle continues until the best set of parameters, meeting the user's requirements, is found. Thus, the optimization algorithm seeks optimal parameters that specifically satisfy the user's needs, defined through the fitness function in "EA Virt".
Indicator Virtualization
To run an advisor on historical data, it is necessary to create a virtual copy of the trading strategy that will execute the same trading operations as when working on a trading account. When indicators are not in use, virtualizing logical conditions within the advisor becomes relatively straightforward; it only requires describing logical actions according to the time point on the price series. While using indicators poses a more complex task, and in most cases, trading strategies rely on the use of various indicators.
The issue arises when searching for optimal indicator parameters, as it requires creating indicator handles with the current set at a given iteration. After running on historical data, these handles must be deleted; otherwise, the computer's RAM can quickly fill up, especially if there are a large number of potential parameter sets. This is not a problem if this procedure is carried out on a symbol chart, but in the tester, handle deletion is not allowed.
To address this issue, we need to "virtualize" the calculation of the indicator within the executing advisor to avoid using handles. Let's take the Stochastic indicator as an example.
The calculation part of each indicator contains a standard function called "OnCalculate". This function needs to be renamed, for example, to "Calculate" and left almost unchanged.
The indicator should be structured as a class (a structure will also work), let's name it "C_Stochastic". In the class declaration, the main indicator buffers should be defined as public fields (additional calculation buffers can be private), and an initialization function "Init" should be declared, where the indicator parameters need to be passed.
//—————————————————————————————————————————————————————————————————————————————— class C_iStochastic { public: void Init (const int InpKPeriod, // K period const int InpDPeriod, // D period const int InpSlowing) // Slowing { inpKPeriod = InpKPeriod; inpDPeriod = InpDPeriod; inpSlowing = InpSlowing; } public: int Calculate (const int rates_total, const int prev_calculated, const double &high [], const double &low [], const double &close []); //--- indicator buffers public: double ExtMainBuffer []; public: double ExtSignalBuffer []; private: double ExtHighesBuffer []; private: double ExtLowesBuffer []; private: int inpKPeriod; // K period private: int inpDPeriod; // D period private: int inpSlowing; // Slowing }; //——————————————————————————————————————————————————————————————————————————————
And, accordingly, the actual calculation of the indicator in the "Calculate" method. The calculation of the indicator is absolutely no different from the indicator in the standard terminal setup. The only difference is the allocation of size for the indicator buffers and their initialization.
This is a very simple example to understand the principle of indicator virtualization. The calculation is performed over the entire depth of periods specified in the indicator parameters.
//—————————————————————————————————————————————————————————————————————————————— int C_iStochastic::Calculate (const int rates_total, const int prev_calculated, const double &high [], const double &low [], const double &close []) { if (rates_total <= inpKPeriod + inpDPeriod + inpSlowing) return (0); ArrayResize (ExtHighesBuffer, rates_total); ArrayResize (ExtLowesBuffer, rates_total); ArrayResize (ExtMainBuffer, rates_total); ArrayResize (ExtSignalBuffer, rates_total); ArrayInitialize (ExtHighesBuffer, 0.0); ArrayInitialize (ExtLowesBuffer, 0.0); ArrayInitialize (ExtMainBuffer, 0.0); ArrayInitialize (ExtSignalBuffer, 0.0); int i, k, start; start = inpKPeriod - 1; if (start + 1 < prev_calculated) { start = prev_calculated - 2; Print ("start ", start); } else { for (i = 0; i < start; i++) { ExtLowesBuffer [i] = 0.0; ExtHighesBuffer [i] = 0.0; } } //--- calculate HighesBuffer[] and ExtHighesBuffer[] for (i = start; i < rates_total && !IsStopped (); i++) { double dmin = 1000000.0; double dmax = -1000000.0; for (k = i - inpKPeriod + 1; k <= i; k++) { if (dmin > low [k]) dmin = low [k]; if (dmax < high [k]) dmax = high [k]; } ExtLowesBuffer [i] = dmin; ExtHighesBuffer [i] = dmax; } //--- %K start = inpKPeriod - 1 + inpSlowing - 1; if (start + 1 < prev_calculated) start = prev_calculated - 2; else { for (i = 0; i < start; i++) ExtMainBuffer [i] = 0.0; } //--- main cycle for (i = start; i < rates_total && !IsStopped (); i++) { double sum_low = 0.0; double sum_high = 0.0; for (k = (i - inpSlowing + 1); k <= i; k++) { sum_low += (close [k] - ExtLowesBuffer [k]); sum_high += (ExtHighesBuffer [k] - ExtLowesBuffer [k]); } if (sum_high == 0.0) ExtMainBuffer [i] = 100.0; else ExtMainBuffer [i] = sum_low / sum_high * 100; } //--- signal start = inpDPeriod - 1; if (start + 1 < prev_calculated) start = prev_calculated - 2; else { for (i = 0; i < start; i++) ExtSignalBuffer [i] = 0.0; } for (i = start; i < rates_total && !IsStopped (); i++) { double sum = 0.0; for (k = 0; k < inpDPeriod; k++) sum += ExtMainBuffer [i - k]; ExtSignalBuffer [i] = sum / inpDPeriod; } //--- OnCalculate done. Return new prev_calculated. return (rates_total); } //——————————————————————————————————————————————————————————————————————————————
Strategy Virtualization
Having discussed the virtualization of the indicator within the advisor, we now move on to considering the virtualization of the strategy. At the beginning of the advisor's code, we declare the import of libraries, including files from the standard trading library and the virtual stochastic file.
Next come the "input" parameters of the advisor, among which we highlight "InpKPeriod_P" and "InpUpperLevel_P". These parameters need to be optimized, representing the period of the "Stochastic" indicator and its levels.
input string InpKPeriod_P = "18|9|3|24"; //STO K period: it is necessary to optimize input string InpUpperLevel_P = "96|88|2|98"; //STO upper level: it is necessary to optimize
Additionally, it is worth noting that the parameters are declared with a string type. These parameters are composite and include default values, the starting optimization value, the step, and the final optimization value.
In the initialization of the advisor within the "OnInit" function, we will set the size of the parameter arrays according to the number of parameters to be optimized: "Set" - the set of parameters, "Range_Min" - the minimum parameter values (starting values), "Range_Step" - the parameter steps, and "Range_Max" - the maximum parameter values. We will extract the corresponding values from the string parameters and assign them to the arrays.
//—————————————————————————————————————————————————————————————————————————————— #import "\\Market\\AO Core.ex5" bool Init (int colonySize, double &range_min [], double &range_max [], double &range_step []); //------------------------------------------------------------------------------ void Preparation (); void GetVariantCalc (double &variant [], int pos); void SetFitness (double value, int pos); void Revision (); //------------------------------------------------------------------------------ void GetVariant (double &variant [], int pos); double GetFitness (int pos); #import //—————————————————————————————————————————————————————————————————————————————— #include <Trade\Trade.mqh>; #include "cStochastic.mqh" input group "==== GENERAL ===="; sinput long InpMagicNumber = 132516; //Magic Number sinput double InpLotSize = 0.01; //Lots input group "==== Trading ===="; input int InpStopLoss = 1450; //Stoploss input int InpTakeProfit = 1200; //Takeprofit input group "==== Stochastic ==|value|start|step|end|=="; input string InpKPeriod_P = "18|9|3|24"; //STO K period : it is necessary to optimize input string InpUpperLevel_P = "96|88|2|98"; //STO upper level: it is necessary to optimize input group "====Self-optimization===="; sinput bool SelfOptimization = true; sinput int InpBarsOptimize = 18000; //Number of bars in the history for optimization sinput int InpBarsReOptimize = 1440; //After how many bars, EA will reoptimize sinput int InpPopSize = 50; //Population size sinput int NumberFFlaunches = 10000; //Number of runs in the history during optimization sinput int Spread = 10; //Spread MqlTick Tick; CTrade Trade; C_iStochastic IStoch; double Set []; double Range_Min []; double Range_Step []; double Range_Max []; double TickSize = 0.0; /—————————————————————————————————————————————————————————————————————————————— int OnInit () { TickSize = SymbolInfoDouble (_Symbol, SYMBOL_TRADE_TICK_SIZE); ArrayResize (Set, 2); ArrayResize (Range_Min, 2); ArrayResize (Range_Step, 2); ArrayResize (Range_Max, 2); string result []; if (StringSplit (InpKPeriod_P, StringGetCharacter ("|", 0), result) != 4) return INIT_FAILED; Set [0] = (double)StringToInteger (result [0]); Range_Min [0] = (double)StringToInteger (result [1]); Range_Step [0] = (double)StringToInteger (result [2]); Range_Max [0] = (double)StringToInteger (result [3]); if (StringSplit (InpUpperLevel_P, StringGetCharacter ("|", 0), result) != 4) return INIT_FAILED; Set [1] = (double)StringToInteger (result [0]); Range_Min [1] = (double)StringToInteger (result [1]); Range_Step [1] = (double)StringToInteger (result [2]); Range_Max [1] = (double)StringToInteger (result [3]); IStoch.Init ((int)Set [0], 1, 3); // set magicnumber to trade object Trade.SetExpertMagicNumber (InpMagicNumber); //--- return (INIT_SUCCEEDED); } //——————————————————————————————————————————————————————————————————————————————Additionally, in the advisor's code within the "OnTick" function, we insert a block calling the self-optimization - the "Optimize" function, which acts as the "manager" in the diagram in Figure 1, initiating the optimization process. Where external variables that need to be optimized were supposed to be used, we utilize values from the "Set" array.
//—————————————————————————————————————————————————————————————————————————————— void OnTick () { //---------------------------------------------------------------------------- if (!IsNewBar ()) { return; } //---------------------------------------------------------------------------- if (SelfOptimization) { //-------------------------------------------------------------------------- static datetime LastOptimizeTime = 0; datetime timeNow = iTime (_Symbol, PERIOD_CURRENT, 0); datetime timeReop = iTime (_Symbol, PERIOD_CURRENT, InpBarsReOptimize); if (LastOptimizeTime <= timeReop) { LastOptimizeTime = timeNow; Print ("-------------------Start of optimization----------------------"); Print ("Old set:"); ArrayPrint (Set); Optimize (Set, Range_Min, Range_Step, Range_Max, InpBarsOptimize, InpPopSize, NumberFFlaunches, Spread * SymbolInfoDouble (_Symbol, SYMBOL_TRADE_TICK_SIZE)); Print ("New set:"); ArrayPrint (Set); IStoch.Init ((int)Set [0], 1, 3); } } //---------------------------------------------------------------------------- if (!SymbolInfoTick (_Symbol, Tick)) { Print ("Failed to get current symbol tick"); return; } //data preparation------------------------------------------------------------ MqlRates rates []; int dataCount = CopyRates (_Symbol, PERIOD_CURRENT, 0, (int)Set [0] + 1 + 3 + 1, rates); if (dataCount == -1) { Print ("Data get error"); return; } double hi []; double lo []; double cl []; ArrayResize (hi, dataCount); ArrayResize (lo, dataCount); ArrayResize (cl, dataCount); for (int i = 0; i < dataCount; i++) { hi [i] = rates [i].high; lo [i] = rates [i].low; cl [i] = rates [i].close; } int calc = IStoch.Calculate (dataCount, 0, hi, lo, cl); if (calc <= 0) return; double buff0 = IStoch.ExtMainBuffer [ArraySize (IStoch.ExtMainBuffer) - 2]; double buff1 = IStoch.ExtMainBuffer [ArraySize (IStoch.ExtMainBuffer) - 3]; //---------------------------------------------------------------------------- // count open positions int cntBuy, cntSell; if (!CountOpenPositions (cntBuy, cntSell)) { Print ("Failed to count open positions"); return; } //---------------------------------------------------------------------------- // check for buy if (cntBuy == 0 && buff1 <= (100 - (int)Set [1]) && buff0 > (100 - (int)Set [1])) { ClosePositions (2); double sl = NP (Tick.bid - InpStopLoss * TickSize); double tp = NP (Tick.bid + InpTakeProfit * TickSize); Trade.PositionOpen (_Symbol, ORDER_TYPE_BUY, InpLotSize, Tick.ask, sl, tp, "Stochastic EA"); } //---------------------------------------------------------------------------- // check for sell if (cntSell == 0 && buff1 >= (int)Set [1] && buff0 < (int)Set [1]) { ClosePositions (1); double sl = NP (Tick.ask + InpStopLoss * TickSize); double tp = NP (Tick.ask - InpTakeProfit * TickSize); Trade.PositionOpen (_Symbol, ORDER_TYPE_SELL, InpLotSize, Tick.bid, sl, tp, "Stochastic EA"); } } //——————————————————————————————————————————————————————————————————————————————Similarly, in the "Optimize" function, the same actions are performed as typically seen in optimization algorithm testing scripts in the series of articles on "Population Optimization Algorithms":
1. Initialization of the optimization algorithm.
2.1. Preparation of the population.
2.2. Obtaining a set of parameters from the optimization algorithm.
2.3. Calculating the fitness function with the parameters passed to it.
2.4. Updating the best solution.
2.5. Obtaining the best solution from the algorithm.
//—————————————————————————————————————————————————————————————————————————————— void Optimize (double &set [], double &range_min [], double &range_step [], double &range_max [], const int inpBarsOptimize, const int inpPopSize, const int numberFFlaunches, const double spread) { //---------------------------------------------------------------------------- double parametersSet []; ArrayResize(parametersSet, ArraySize(set)); //---------------------------------------------------------------------------- int epochCount = numberFFlaunches / inpPopSize; Init(inpPopSize, range_min, range_max, range_step); // Optimization------------------------------------------------------------- for (int epochCNT = 1; epochCNT <= epochCount && !IsStopped (); epochCNT++) { Preparation (); for (int set = 0; set < inpPopSize; set++) { GetVariantCalc (parametersSet, set); SetFitness (VirtualStrategy (parametersSet, inpBarsOptimize, spread), set); } Revision (); } Print ("Fitness: ", GetFitness (0)); GetVariant (parametersSet, 0); ArrayCopy (set, parametersSet, 0, 0, WHOLE_ARRAY); } //——————————————————————————————————————————————————————————————————————————————
Additionally, the "VirtualStrategy" function conducts strategy testing on historical data (referred to as "EA Virt" in Figure 1). It takes an array of parameters "set", the number of bars for optimization "barsOptimize", and the "spread" value.
The data preparation phase comes first. Historical data is loaded into the "rates" array. Arrays "hi", "lo", and "cl" are then created, necessary for calculating Stochastic.
Next, the Stochastic indicator is initialized, and its calculation is performed based on historical data. If the calculation fails, the function returns "-DBL_MAX" (the worst possible fitness function value).
Subsequently, the strategy is tested on historical data, following the logic identical to the main advisor's code. A "deals" object is created to store trades. A loop iterates through historical data, where conditions for opening and closing positions are checked for each bar based on the indicator value and "upLevel" and "dnLevel" levels. If conditions are met, positions are opened or closed.
After iterating through historical data, the function checks the number of trades executed. If no trades were made, the function returns "-DBL_MAX". Otherwise, it returns the final balance.
The return value of "VirtualStrategy" represents the fitness function value. In this case, it is the final balance in points (as mentioned earlier, the fitness function could be the balance, profit factor, or any other metric indicating the quality of trading results on historical data).
It is important to note that the virtual strategy must closely match the advisor's strategy. In this example, trading is based on opening prices, corresponding to the bar opening control in the main advisor. If the trading strategy logic operates on every tick, the user needs to ensure tick data is available during virtual testing and adjust the "VirtualStrategy" function accordingly.
//—————————————————————————————————————————————————————————————————————————————— double VirtualStrategy (double &set [], int barsOptimize, double spread) { //data preparation------------------------------------------------------------ MqlRates rates []; int dataCount = CopyRates(_Symbol, PERIOD_CURRENT, 0, barsOptimize + 1, rates); if (dataCount == -1) { Print ("Data get error"); return -DBL_MAX; } double hi []; double lo []; double cl []; ArrayResize (hi, dataCount); ArrayResize (lo, dataCount); ArrayResize (cl, dataCount); for (int i = 0; i < dataCount; i++) { hi [i] = rates [i].high; lo [i] = rates [i].low; cl [i] = rates [i].close; } C_iStochastic iStoch; iStoch.Init ((int)set [0], 1, 3); int calc = iStoch.Calculate (dataCount, 0, hi, lo, cl); if (calc <= 0) return -DBL_MAX; //============================================================================ //test of strategy on history------------------------------------------------- S_Deals deals; double iStMain0 = 0.0; double iStMain1 = 0.0; double upLevel = set [1]; double dnLevel = 100.0 - set [1]; double balance = 0.0; //running through history----------------------------------------------------- for (int i = 2; i < dataCount; i++) { if (i >= dataCount) { deals.ClosPos (-1, rates [i].open, spread); deals.ClosPos (1, rates [i].open, spread); break; } iStMain0 = iStoch.ExtMainBuffer [i - 1]; iStMain1 = iStoch.ExtMainBuffer [i - 2]; if (iStMain0 == 0.0 || iStMain1 == 0.0) continue; //buy------------------------------- if (iStMain1 <= dnLevel && dnLevel < iStMain0) { deals.ClosPos (-1, rates [i].open, spread); if (deals.GetBuys () == 0) deals.OpenPos (1, rates [i].open, spread); } //sell------------------------------ if (iStMain1 >= upLevel && upLevel > iStMain0) { deals.ClosPos (1, rates [i].open, spread); if (deals.GetSels () == 0) deals.OpenPos (-1, rates [i].open, spread); } } //---------------------------------------------------------------------------- if (deals.histSelsCNT + deals.histBuysCNT <= 0) return -DBL_MAX; return deals.balance; } //——————————————————————————————————————————————————————————————————————————————
