Walk-forward optimization library for MetaTrader: pause, resume, refine

Version 1.12 of WFO-library introduces an interesting and powerful feature - a possibility to pause and resume an optimization. Until now WFO-library supported only separate uninterrupted optimizations.

From the point of view of EA developer, new mode can be enabled by new flag WFO_FLAG_RESUME_OPTIMIZATION (32) passed to the function wfo_setAdvancedOptions. But apart from the coding, the feature requires some special procedure for proper optimization setup. Here I'll provide some instructions. Yet before we begin, it's important to refresh your knowledge on how optimization process is organized in MetaTrader 5 technically.

Theory

The tester supports 2 types of optimization:

• slow optimization with complete iteration through all combinations of input parameters, and
• fast optimization based on genetic algorithm;

Genetics speed up the process via selection of a small part of possible combinations - of course the selection is not random but backed up with a sophisticated logic (lying far beyond the scope of this short publication), invented for finding quasi-optimal solution with best ratio of "optimality" and time spent to find it.

I remind you that even if you select slow optimization, the tester may automatically switch to genetics if the number of combinations is too large (for example, over 100 mln in case of using 64-bit terminal).

Every optimization produces a cache file - an opt-file located in the subfolder /tester/cache/ of your MT5 folder. The name of the file starts with a prefix equal to EA name and also includes work symbol, timeframe, date range, and a hash number, uniquely identifying current "EA build" and settings. For example,

WalkForwardDemo.EURUSD.H1.20230101.20231026.11.EF7BF3C92D7F53F512CC1E61A72B9448.opt

What is the "EA build" - I'll describe a few lines below.

Now it's important to note, that the cache can be used not only to review results of recent optimizations just in the tester UI, but to pause and resume a lengthy optimization. Indeed, as soon as you start an optimization, the tester gathers incoming results in the opt-file, which is sufficient to continue the process with respect to already made efforts. You may stop optimization at any moment, and then resume it by pressing Start at any time later, given that the opt-file still exists, and the hash matches "EA build" and settings.

"EA build" is a hash of binary ex5-file (and all binary libraries, if they are used, such as WalkForwardOptimizer.ex5). Every time you recompile EA (or replace a library) the hash is changed, which invalidates the opt-file cache.

Please note, that MetaEditor generates a unique ex5-file as a result of any compilation, even if the source code was not changed. This is a part of protection against decompilation.

Changes in the tester settings or input parameters will also make opt-file cache inconsistent.

Unfortunately, there is no way to discover which ex5-file variant and settings are encoded in the hash (hash calculation is a one-way algorithm). So, it's a bit difficult for a user to distinguish one opt-cache from the other and figure out if it still corresponds for the current environment, and hence valid for optimization resumption. The only attribute a user can use for investigation is the timestamp of the opt-file (last time optimization data was updated). A user should make a note somewhere, which settings and inputs were used for specific opt-cache (marked by specific date/time).

Actually a user can select caches one by one in the tester UI from the dropdown list with all cached optimizations, read their settings and inputs, but UI does not provide the opt-file names, so if there are many caches for the same EA, symbol, timeframe and date ranges, the user need somehow deduce, which hash corresponds to optimization run of his/her interest.

If the cache became invalid (as a result of changes in at least one of the elements: ex5-file, or settings, or inputs), next time you start an optimization it will be a new optimization, not a continuation of a previous one. If ex5-file was changed, the tester removes opt-file (if exists for current settings/inputs) and outputs a message about this into the log.

For genetic optimizations there exists even more interesting possibility. After a genetic optimization is finished, you can restart it and continue the process, taking into account all cached results, that is not from scratch.

You can restart genetic optimization again and again. Every restart will most likely find more and more better results, all summed up in the opt-file.

This works smoothly until you don't customize optimization in any peculiar way, such as WFO. The problem arises from several aspects.

MQL5 API does not provide any means to let MQL-program know, if current optimization is a new one or a continuation based on pre-existing opt-cache. In other words, neither your EA nor WFO library can automatically detect whether to gather their specific data (complementing standard optimization data) to a new file or append to an old one (if it exists). Moreover, it's impossible to figure out, if a valid opt-cache exists for current "EA build" and settings. And moreover, it's impossible to detect a state of optimization according to an existing opt-file: it may belong to a finished slow optimization (it can't be resumed), to a paused optimization (no matter slow or fast, both can be resumed), or to a finished genetic - the latter can be restarted/refined. Strictly speaking there are some "hacks", which are not parts of MQL5 API, and allowing to read opt-files - they are insufficient and unreliable for use in a product for MQL5 market.

This is why WFO library did not allow optimization suspension/continuation until version 1.12. The library starts collecting its specific data from very beginning upon each optimization start. It implies that if a valid opt-file exists, it should have been removed manually by a user before new optimization.

Practice

Starting from version 1.12 you may set the flag WFO_FLAG_RESUME_OPTIMIZATION in your code. As a result WFO library will keep existing WFO data during optimization start and append new passes into it, that is in sync with filling opt-file by the tester.

To enable/disable the flag one can use the following approach:

#include <WalkForwardOptimizer.mqh>
sinput ulong wfo_advancedOptions = 0; // wfo_advancedOptions (bitwise flags)

int OnInit()
{
  ...
  wfo_setAdvancedOptions(wfo_advancedOptions); // must be called before wfo_OnInit
  return wfo_OnInit(wfo_windowSize, wfo_stepSize, wfo_stepOffset, wfo_customWindowSizeDays, wfo_customStepSizePercent);
}

void OnTesterInit()
{
  ...
  wfo_setAdvancedOptions(wfo_advancedOptions); // must be called before wfo_OnTesterInit
  wfo_OnTesterInit(wfo_outputFile); // required
}

The function wfo_setAdvancedOptions is called twice in OnInit and OnTesterInit for generality here, because different flags are used in different contexts: part of the flags are applicable on the agents, and part of the flags - in the terminal (see https://www.mql5.com/en/blogs/post/754712).

Specifically the flag WFO_FLAG_RESUME_OPTIMIZATION (32) makes sense in OnTesterInit only.

Don't forget that changing wfo_advancedOptions input itself changes the hash and invalidates opt-cache (if it exists). Be careful:

• when you change 0 to 32 in wfo_advancedOptions, make sure there is no WFO-related files, because the tester should normally create a new opt-file;
• yet if you did already use 32 some day ago, an opt-file corresponding to these inputs may still exist - then you need to make sure (check your side notes, for example) that proper WFO-related files do also exist;
• when you change 32 to 0 in wfo_advancedOptions, make sure there is no an opt-file suitable for continuation, because WFO-files will be deleted;

If you are absolutely sure that you always want to continue optimization (that is WFO-related files, such as csv and gvf, if they exist, correspond to an opt-file eligible for continuation), then you can hardcode the new mode in the following way:

#include <WalkForwardOptimizer.mqh>

void OnTesterInit()
{
  ...
  wfo_setAdvancedOptions(WFO_FLAG_RESUME_OPTIMIZATION);
  wfo_OnTesterInit(wfo_outputFile);
}

It's also safe approach, if there is no caches at all (neither a suitable opt-file, nor WFO-related files) - you should check this (or clean up) manually before new optimization.

The new mode is in particular useful for multiple successive runs of genetic optimizations. Because genetic optimization selects very limited number of passes from entire optimization space, it produces WFO-reports with a lot of holes in forward steps (remember - steps are organized by increments of wfo_stepOffset, and genetics may consider to probe only a small part of the steps).