MQL's OOP notes: On The Fly Self-Optimization of Expert Advisers: part 2
The library source code resides in the file Optimystic.mq4, and it's currently almost empty: in the part 1 we've left it on:
#property strict
#define OPTIMYSTIC_LIBRARY
#include <Optimystic.mqh>
The only function that is exported from the library is the factory:
{
return NULL; // ???
}
It should return an object which implements Optimystic interface. Let's start coding it.
{
private:
OptimysticCallback *callback;
bool enabled;
int rangeInBars;
int resetInBars;
PERFORMANCE_ESTIMATOR estimator;
double deposit;
int spread;
double commission;
public:
OptimysticImplementation(OptimysticCallback &cb)
{
callback = GetPointer(cb);
}
virtual void setEnabled(const bool _enabled)
{
enabled = _enabled;
}
virtual OptimysticParameter * addParameter(const string name, const double min, const double max, const double step)
{
return NULL;
}
virtual OptimysticParameter *getParameter(const int i) const
{
return NULL;
}
virtual void setHistoryDepth(const int size, const TIME_UNIT unit)
{
if(unit == UNIT_DAY)
{
rangeInBars = size * (PeriodSeconds(PERIOD_D1) / PeriodSeconds());
}
else
{
rangeInBars = size;
}
}
virtual void setReoptimizationPeriod(const int size, const TIME_UNIT unit)
{
if(unit == UNIT_DAY)
{
resetInBars = size * (PeriodSeconds(PERIOD_D1) / PeriodSeconds());
}
else
{
resetInBars = size;
}
}
virtual void setOptimizationCriterion(const PERFORMANCE_ESTIMATOR type)
{
estimator = type;
}
virtual void setDeposit(const double d)
{
deposit = d;
}
virtual void setSpread(const int points = 0)
{
spread = points;
}
virtual void setCommission(const double c)
{
commission = c;
}
virtual void onTick()
{
}
virtual double getOptimalPerformance(PERFORMANCE_ESTIMATOR type = ESTIMATOR_DEFAULT)
{
return 0;
}
};
We're currently missing another class - a class with an implementation of OptimysticParameter. Let's code it as well.
{
protected:
string name;
double min;
double max;
double step;
double value;
double best;
bool enabled;
public:
OptimysticParameterImplementation(const string _name, const double _min, const double _max, const double _step)
{
name = _name;
min = MathMin(_min, _max);
max = MathMax(_max, _min);
enabled = true;
step = _step <= 0 ? 1 : _step;
best = EMPTY_VALUE;
}
void setValue(const double v)
{
value = v;
}
void increment()
{
value += step;
}
void markAsBest()
{
best = value;
}
virtual string getName() const
{
return name;
}
virtual double getMin() const
{
return min;
}
virtual double getMax() const
{
return max;
}
virtual double getStep() const
{
return step;
}
virtual double getValue() const
{
return value;
}
virtual double getBest() const
{
return best;
}
virtual bool getEnabled() const
{
return enabled;
}
virtual void setEnabled(const bool e)
{
enabled = e;
}
};
To store array of parameters inside OptimysticImplementation I'll use helper class RubbArray (it was described in the blog ealier, but also attached below).
{
private:
RubbArray<OptimysticParameterImplementation> parameters;
int parameterSpace[];
int parameterCursor[];
double best;
Now we can complete implementation of the methods addParameter and getParameter:
{
parameters << new OptimysticParameterImplementation(name, min, max, step);
return parameters.top();
}
virtual OptimysticParameter *getParameter(const int i) const
{
return parameters[i];
}
Now we can focus on the main method onTick. We can't write it at once, so start with a schematic draft.
{
static datetime lastBar;
if(enabled)
{
if(lastBar != Time[0] && TimeCurrent() > lastOptimization + PeriodSeconds() * resetInBars)
{
callback.setContext(context); // enable virtual trading
optimize();
}
}
callback.setContext(NULL); // enable real trading
callback.trade(0, Ask, Bid); // actual trading
lastBar = Time[0];
}
This will not compile, because some things here are undefined yet. For example, we need a private variable lastOptimization in the class to store a time of last optimization. We will assign it in the private method optimize, which should be also added to perform actual work. And we don't yet know what is the context. Let's fix the first 2 points:
datetime lastOptimization;
protected:
bool optimize()
{
// ...
lastOptimization = TimeCurrent();
return true;
}
As for the context, it should be a variable with implementation of the interface OptimysticTradeContext, which is left the only unimplemented one.
{
// ...
};
It must contain a lot of methods. You can just copy and paste the declaration of the interface into the class and code all methods scrupulously, one by one. Here, in the blog, it's not possible to cover all the changes, so please consult with the attached source code to get the final picture.
{
private:
int bar;
public:
OptimysticTradeContextImplementation()
{
}
void setBar(const int b)
{
bar = b;
}
If you remember, we have a method optimize in OptimysticImplementation. This is where we place a cycle through bars on backtest period, and it will call setBar. Something like this:
{
protected:
bool optimize()
{
if(!callback.onBeforeOptimization()) return false;
Print("Starting on-the-fly optimization at ", TimeToString(Time[rangeInBars]));
if(spread == 0) spread = (int)MarketInfo(Symbol(), MODE_SPREAD);
/* TODO: init parameters */
do
{
if(!callback.onApplyParameters()) return false;
// build balance and equity curves on the fly
for(int i = rangeInBars; i >= 0; i--)
{
context.setBar(i);
callback.trade(i, iOpen(Symbol(), Period(), i) + spread * Point, iOpen(Symbol(), Period(), i));
}
// TODO: calculate performance
callback.onReadyParameters();
}
while(/* TODO: loop through possible parameter combinations */);
callback.onAfterOptimization();
lastOptimization = TimeCurrent();
return true;
}
Having the current bar in the context we can implement specific methods, for example, TimeCurrent:
{
public:
virtual datetime TimeCurrent()
{
return iTime(::Symbol(), ::Period(), bar);
}
This is the first method of the abstract interface OptimysticTradeContext which got a concrete implementation in OptimysticTradeContextImplementation. Some of the other methods can be implemented in the similar way:
{
int b = bar + offset;
if(symbol != Symbol())
{
b = ::iBarShift(symbol, tf, Time[b]);
}
return ::iOpen(symbol, tf, b);
}
But the others require additional work to be done. For example, before we can create a new order with OrderSend we surely need to define a special class.
{
private:
static int tickets;
public:
int type;
string symbol;
double openPrice;
datetime openTime;
double lots;
double takeProfit;
double stopLoss;
string comment;
datetime expiration;
int magicNumber;
color arrow;
int ticket;
double closePrice;
datetime closeTime;
Order(datetime now, string s, int cmd, double volume, double price, int deviation, double stoploss, double takeprofit, string cmnt = NULL, int magic = 0, datetime exp = 0, color clr = clrNONE)
{
symbol = s;
type = cmd;
lots = volume;
openPrice = price;
openTime = now;
stopLoss = stoploss;
takeProfit = takeprofit;
comment = cmnt;
magicNumber = magic;
expiration = exp;
arrow = clr;
ticket = ++tickets;
}
The context should maintain an array of all market and history orders. So let's add them using RubbArray again:
{
private:
RubbArray<Order> orders;
RubbArray<Order> history;
And implement OrderSend:
{
Order *order = new Order(TimeCurrent(), symbol, cmd, volume, price, deviation, stoploss, takeprofit, comment, magic, expiration, arrow);
orders << order;
return order.ticket;
}
To close an order we could code something like this:
{
Order *order = findByTicket(ticket, orders);
if(order != NULL && (order.type == OP_BUY || order.type == OP_SELL))
{
orders >> ticket; // remove from open orders
history << order; // and add it to the history
order.closePrice = price;
order.closeTime = TimeCurrent();
// TODO: calculate profit and other statistics
return true;
}
return false;
}
We use new private helper here - findByTicket. It's code is more or less obvious:
{
int n = array.size();
for(int i = 0; i < n; i++)
{
Order *order = array[i];
if(order.ticket == ticket)
{
ticket = i;
return order;
}
}
return NULL;
}
The calculation of order's profit (that shown as TODO in OrderClose) can be coded inside the Order class itself:
{
public:
double profit(double price, const int spread)
{
if(type == OP_BUY || type == OP_SELL)
{
if(closePrice != 0) price = closePrice;
double ptv = MarketInfo(symbol, MODE_POINT) * MarketInfo(symbol, MODE_TICKVALUE) / MarketInfo(symbol, MODE_TICKSIZE);
double pts = (type == OP_BUY ? +1 : -1) * (price - openPrice) / MarketInfo(symbol, MODE_POINT);
return (pts - spread) * ptv * lots;
}
return 0;
}
Now we can calculate profit in OrderClose. The profit should update balance curve, which we don't have yet. Actually we need to collect many more statistics about trading, such as order counter, equity curve, etc. Let's add them to the context:
{
private:
RubbArray<BarState> income;
double profit;
double plusProfit;
double minusProfit;
int plusCount;
int minusCount;
Unfortunately we use yet another new class here - BatState. The good news is that this is the last class in our implementation. And it's simple.
{
public:
datetime timestamp;
double balance;
double equity;
BarState(const datetime dt, double b): timestamp(dt), balance(b), equity(b) {}
BarState(const datetime dt, double b, double e): timestamp(dt), balance(b), equity(e) {}
};
Get back to the context, and we can finally complete OrderClose (only profit-related part is shown):
income.top().balance += amount;
if(amount > 0)
{
plusProfit += amount;
plusCount++;
}
else
{
minusProfit -= amount;
minusCount++;
}
You may see how we update balance and the counters on current bar. Of course, they should be initalized somewhere. So we need to add such a method in the context:
{
public:
void initialize(const double d, const int s, const double c)
{
deposit = d;
spread = s;
commission = c;
plusProfit = minusProfit = 0;
plusCount = minusCount = 0;
income.clear();
income << new BarState(TimeCurrent(), deposit);
}
And it should be called from OptimysticImplementation::optimize method before we run the cycle through history bars. All input values are already known there.
if(!callback.onApplyParameters()) return false;
context.initialize(deposit, spread, commission);
// build balance and equity curves on the fly
for(int i = rangeInBars; i >= 0; i--)
{
context.setBar(i);
callback.trade(i, iOpen(Symbol(), Period(), i) + spread * Point, iOpen(Symbol(), Period(), i));
context.activatePendingOrdersAndStops();
context.calculateEquity();
}
context.closeAll();
// calculate performance
context.calculateStats();
callback.onReadyParameters();
In addition to initialize you see a bunch of other new methods called on the context. I hope their names are self-explaining. We need to implement them all, but let us skip details here - you may consult with the source code. The only thing which can fit is a note about income array. We need to add new element into it on every bar, this is why we extend the function setBar a bit:
{
bar = b;
income << new BarState(TimeCurrent(), income.top().balance);
}
This code simply copies last known balance to the new bar, and if some orders will be closed (as you saw in OrderClose), their profit will update the balance.
{
return income.size() > 0 ? income.top().balance : 0;
}
As soon as we have total profit and other stats after the loop through the bars, the context can calculate performance indicators. So let's add a method which return them.
{
if(type == ESTIMATOR_PROFIT_FACTOR)
{
return minusProfit != 0 ? plusProfit / minusProfit : DBL_MAX;
}
else if(type == ESTIMATOR_WIN_PERCENT)
{
return plusCount + minusCount != 0 ? 100.0 * plusCount / (plusCount + minusCount) : 0;
}
else if(type == ESTIMATOR_AVERAGE_PROFIT)
{
return plusCount + minusCount != 0 ? AccountBalance() / (plusCount + minusCount) : 0;
}
else if(type == ESTIMATOR_TRADES_COUNT)
{
return plusCount + minusCount;
}
// default: (type == ESTIMATOR_DEFAULT || type == ESTIMATOR_NET_PROFIT)
return AccountBalance() - deposit;
}
This method can be used to provide implementation for Optimystic::getOptimalPerformance, which is very important for the client EA.
With the help of getPerformance optimization process (in OptimysticImplementation::optimize) can be refined further.
/* TODO: resetParameters(); */
do
{
// loop through bars goes here
...
// calculate performance (please find details in the sources)
context.calculateStats();
double gain = context.getPerformance(estimator);
// if estimator is better than previous, then save current parameters
if(gain > best && context.getPerformance(ESTIMATOR_TRADES_COUNT) > 0)
{
best = gain;
/* TODO: storeBestParameters(); call markAsBest for every parameter */
}
}
while(/* TODO: iterateParameters() */);
All calls related to the parameter set enumeration and selection are still marked as TODO. Due to the lack of space their implementation is not covered here and can be studied in the source code.
And that's actually all. The library is ready. It works.
- incomplete multicurrency support: specifically we need to have separate spread values for every symbol;
- no swaps calculation;
- no fast optimization by means of genetics or other algorithm (so don't specify too many combinations of parameters to check, because straightforward approach is currently used, and it's slow);
- no validation of function parameters and error generation (such as, invalid stops);
- tick values are taken for current exchange rates, whereas they should be recalculated using historic quotes;
- rough price simulation mode (by open prices);
Expert Adviser Example
After you add the include:
Declare your own class derived from OptimysticCallback:
{
private:
void print(const string title);
public:
virtual bool onBeforeOptimization();
virtual bool onApplyParameters();
virtual void trade(const int bar, const double Ask, const double Bid);
virtual void onReadyParameters();
virtual void onAfterOptimization();
};
and implement all the methods. Part of them was already described in the part 1. Most interesting is that code for the method trade should be taken completely from the former OnTick event handler. The lines that invoke indicators should be changed to take specific bar number passed via the first parameter:
{
double MacdCurrent, MacdPrevious;
double SignalCurrent, SignalPrevious;
double MaCurrent, MaPrevious;
MacdCurrent = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_MAIN, bar + 0);
MacdPrevious = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_MAIN, bar + 1);
SignalCurrent = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_SIGNAL, bar + 0);
SignalPrevious = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_SIGNAL, bar + 1);
MaCurrent = iMA(NULL, 0, MATrendPeriod, 0, MODE_EMA, PRICE_CLOSE, bar + 0);
MaPrevious = iMA(NULL, 0, MATrendPeriod, 0, MODE_EMA, PRICE_CLOSE, bar + 1);
All other calls made in the source to seemingly standard functions are intercepted by OptimysticCallback and redirected to appropriate backend: either the platfrom or our virtual trade context. And even Ask and Bid are replaced with correct values for the bar bar.
void MyOptimysticCallback::onAfterOptimization()
{
if(p.getOptimalPerformance(Estimator) > 0)
{
// apply new parameters for real trading
MACDOpenLevel = p[0].getBest();
MACDCloseLevel = p[1].getBest();
MATrendPeriod = (int)p[2].getBest();
Print("Best parameters: ", (string)MACDOpenLevel, " ", (string)MACDCloseLevel, " ", (string)MATrendPeriod);
TradeEnabled = true;
}
else
{
Print("No optimum");
TradeEnabled = false;
}
}
Then in the method trade:
{
// ...
int cnt, ticket, total;
total = OrdersTotal();
if(total < 1 && ((getContext() != NULL) || TradeEnabled))
{
Hence, new orders can be opened only if virtual context is not NULL (that is in optimization mode) or an applicable parameter set exists (TradeEnabled is true in real trading mode).
Please, note, that since we need to change input parameters from MQL code they should be defined as extern instead of input. Also when on the fly optimization is enabled, input parameters which undergo optimization specify minimal value and step for optimization (they are the same for simplicity), and their maximal value is 10 times larger the minimal value. As a result, if you set 3 parameters for optimization, they will make 1000 combinations.
And here is the final warning: the MACD Sample was taken as example due to its simplicity and availability, but the EA itself is not very good (for example, stoploss is applied on orders only after they go in sufficient profit, so if price moves in unfavourable direction from very beginning, you'll get large loss), as a result you should not expect much from the EA performance. Try the library with your own EAs.
I'm sure there is still much to discuss on the subject, but the blog post can't last for ever, even if it's a very very long story. You may explore the library yourself and try MQL OOP in the action.
