Optimization results are not matching with results from a single test run

 

Hey guys,

My recent optimization results are not what they seem to be. When double-clicking an entry in that table, both backtest and a visualized strategy run yield a totally different result (results of backtest and single visualization test match).

Not only the profit is mismatching, but also the total trades differ (as you see below). So, something is really off here and I don't know why.

My attempts to fix this error so far:

- Thought my hardware with an AMD Ryzen 9300X with its 12x2 hyper-threading cores might be the reason, but I can reproduce this error on a machine with a non-HT CPU (like Intel i7).

- Static variables in my code that could potentially be read by other threads? (Can this actually happen?) -> Anyway, I replaced them all with member variables -> still didn't fix it.

- I'm making use of OnTester events to capture data frames so after all strategies have run the data frames are read and their data placed into a database. The erroneous results are actually the same in the database, so I conclude the number in the 'optimization result' table are not just random numbers, but actual results from properly done simulations, but something must have happened there vastly different.


Optimization Results:


Result from a visualization run:


Has experienced the same discrepancies in the trading results and found out a reason for this?

If you need further details for an educated guess, let me know.

 
Marcel Fitzner:
...

Has experienced the same discrepancies in the trading results and found out a reason for this?

If you need further details for an educated guess, let me know.

Without the code to reproduce it, it's just a (pseudo) random guess.

  • Is it also happening if you choose USD as account currency in the tester ?
  • And if you don't use a template ?
  • What if you delete the cache and run the single backtest again ?
  • Are you using function like GetTickCount() or GetMicrosecondCount() ?
 

Hi Alain,

thanks for looking into this! Interestingly, rerunning those 3456 simulations as often as I want, only the highest profit remains the same (51706.31), while others randomly differ. In a second run there were no more losses shown in the list, on a 3rd rerun they re-occur. So, yes, I agree, mate. There must be some randomness in the whole process.

The trading system I am honing now for several months, was working reliably, up to the point where I must have broken something by re-factoring or restructuring the architecture since the overall performance was not satisfying. However, I did not always "double-click" the optimization results. So at some point it must have been added some random aspect in my code which I am not aware of. Darn, I'd really love to have regression tests to Strategy Testing.

The latest optimization results (with the same 3456 parameter combinations) versus the 2 previous ones:


- I cleared the cache before double clicking a single result, and their profits and # of total trades still differ.

- Removing/Disabling the template (MarginCallCalculator.tpl) did not fix the behaviour.

- Rerunning with USD (instead EUR) results in a similar erroneous behaviour:

The high profits of over 50k are not generated, the best profit results are somewhat around 3.6k and are correct on a double click.
The lossy results are incorrect: double-clicking them results in all being profitable runs.

Heck, what is that??

I thought double-clicking a single result (from above screenshot with the selected line) updates the corresponding parameters in the 'Inputs' tab,
but this is obviously not happening:


Regarding your last question if I am using GetTickCount or GetMicrosecondCount():
Yes, I use GetMicrosecondCount() a couple of times, but they are always guarded by an input parameter which is set to false during testing:

ulong timeOfStart = 0;
        
if (LogPerformance)
   timeOfStart = GetMicrosecondCount();

// ...

if (LogPerformance && !m_StratTesterRunning)
{
   comment = StringFormat("Finished computing buffers. Calculation time: %s ms", DoubleToString((GetMicrosecondCount() - timeOfStart) / 1000.0, 2));
   Print(comment);
}

---

Maybe you or someone else has an idea what may cause that input parameters are not taken over properly.

Thinking back of what I recently did: My last big overhaul was reducing a lot of input parameters, and since I am working with a partner who was confused by a lot of parameters that I intented to keep as "general trading settings for kind-of any situation* - I decided to make a trade-off: No more showing them as input parameters, but keeping them globally with defaults settings - and when I need them for another EA I would not have lost them ;).

Here's my input parameter list of my TradeManager.mqh which is dealing with the overall trading aspects:

sinput group    "TradeManager - Volume:"                                // #############################################################################################
input double    LotSize                                                 = 1;            // Fix lot size when opening an initial position.
input bool      ActivateRiskToBalanceRatioForVol                        = false;        // Activate 'risk to equity' ratio to adjust volume dynamically?
input double    RiskToBalanceRatioForVol                                = 0.005;        // (If above true) use this risked/equity ratio to pick volume dynamically
input bool      ActivateTPIfRewardReachedMultipleRiskDistance           = false;        // Activate Take Profit (uses multiple of risk distance)?
input double    MultipleOfRiskDistanceForTP                             = 1.0;          // Multiple of risk dist. for TP
sinput group    "TradeManager - Leverages:"                             // #############################################################################################
sinput double   MaxAllowedLeverageByESMA                                = 30.0;         // MaxAllowedLeverageByESMA: Fixed value of 30 for private investors.
sinput double   LeverageOfThisSymbol                                    = 20.0;         // LeverageOfThisSymbol: For S&P500 it's 1/20, seek broker's info
sinput double   MarginCallOutLevel                                      = 0.5;          // Margin Call Level by your broker (e.g. 50%)
sinput string   DefaultSymbolForCurrencyConversion                      = "EURUSD";     // Default symbol to convert symbol currency to account currency
input group     "TradeManager - Stop Loss Settings:"                    // #############################################################################################
input PlacementModeOfSLOnNewPos         SLPlacementOnNewPosition        = NO_SL;        // Placement of initial SL after opened position
input TrailingModeOnUpdateSL            SLTrailingMode                  = MANUAL;       // Trailing mode of SL
input TriggerDistanceToUpdateSL         InpTriggerDistanceToUpdateSL    = DISTANCE_2_X_FIX;   // Min. Distance to trigger SL update
input SLUpdateMode                                      InpSLUpdateMode = UPDATE_ON_NEW_BAR;  // Timing for updating an SL
input bool      AutoStopLossOnAutoRepurchase                            = true;         // After AUTO repurchase in profit: Auto Place SL betw. CP & OP?
input bool      KeepInitialSLInLossUntilNewUpdateIsInProfit             = true;         // Keep initial SL in loss until SL update in profit?
input bool      NeverLowerSLInProfits                                   = true;         // If in profits: Never lower SL?
input bool      NeverLowerSLInLosses                                    = true;         // If in losses: Never lower SL?
input double    FixSLDistanceInPercentOfPrice                           = 0.005;        // Fix SL distance (in fraction of price!)
input double    RelSLDistanceBetweenCurrentAndOpenPrice                 = 0.35;         // Rel. SL distance betw. curr. & open price (0.0 - 1.0)
input group     "TradeManager - Auto Re/Purchase Mechanism:"            // ############################################################################
input bool      AutoRepurchaseInProfit                                  = false;        // Auto repurchase in profit position?
input bool      AutoRepurchaseInLosses                                  = false;        // Auto repurchase in loss position?
//input double  MinimumProfitBalanceRatioForRepurchase                  = 0.0005;       // Minimum (profit/balance) ratio for auto repurchase (0.0 - 1.0)
input double    CPToOPDistanceForRepurchaseInPercentOfPrice             = 0.03;         // Auto Repurchase if |Current Price - Open Price| > x% of price
input bool      WhenSLInLossZoneAfterRepurchaseMoveSLAboveFlat          = true;         // On repurchase & SL in loss zone: Move above flat?
input RepurchasingMode RepurchaseMode                                   = CONSTANT_VOLUME;   // Repurchase mode
//input group   "TradeManager - Auto-Repurchase after Stop Loss:"       // #############################################################################################
/*input*/ bool  AutoReOpenAfterStopLoss                                 = false;        // After stop loss: Auto reopen?
/*input*/ ENUM_ORDER_TYPE OrderTypeOnReOpenAfterStopLoss                = ORDER_TYPE_BUY;    // After stop loss: Order type for auto reopen
//input group   "TradeManager - Auto Purchase if Price enters Price Range:" // ############################################################################
/*input*/ bool  IfPriceInPriceRangeAutoInitialPurchase                  = false;        // If price in price range: Auto purchase?
/*input*/ ENUM_ORDER_TYPE IfPriceInPriceRangeOrderTypeOnInitialTrade    = ORDER_TYPE_BUY;   // If price in price range: Order type
/*input*/ double InitialPurchaseIfPriceIsAboveOrEqual                   = DBL_MAX;      // If price in price range: Initial order if price >= x
/*input*/ double InitialPurchaseIfPriceIsBelowOrEqual                   = DBL_MIN;      // If price in price range: Initial order if price <= x
input group     "TradeManager - Prevent Loss of Profit:"                // ############################################################################
input int       MaxNumberOfBarsBeforeAutoStopLoss                       = 20;           // # of Bars before auto SL (uses rel. SL distance), -1 for off!
input double    ProfitBalanceRatioForProfitLossPrevention               = 0.05;         // profit/balance ratio for profit loss prevention (0.0 - 1.0)
input double    ProfitLossToTriggerStopLoss                             = 0.7;          // Amount of lost profit (in %) so that an SL is placed (0.0 - 1.0)
input double    InpSLLevelForProfitLossPrevention                       = 0.7;          // SL level to prevent big profit loss (0.0 - 1.0)
input group     "TradeManager - Max Trades per day:"                    // #############################################################################################
input bool      ActivateMaxTradesPerDay                                 = true;         // Activate max trades per day restriction?
input int       MaxTradesInLossPerDay                                   = 1;            // Max Trades in loss per day - Recommended: 1 (-1 for endless)
input int       MaxTradesInProfitPerDay                                 = -1;           // Max Trades in profit per day (-1 for endless)
sinput group    "TradeManager - Sounds:"                                // ############################################################################
sinput bool     PlaySoundWhenPositionBecomesProfitable                  = true;         // Play Sound when position becomes profitable?
sinput double   PlaySoundOnProfitGainingXPercentOfBalance               = 0.5;          // Play sound when profit gains x% of balance

I would assume these global variables (with /*input*/ ) pose no problem to MetaTrader.

 

You need to determine if the problem comes from your code or from MT5. I would suggest you to check with a "standard" EA like Moving Average or MACD provided with MT5.

Instead of double-clicking if you take the corresponding settings and set them manually, the backtest match the optimization result ?

I have seen a lot of topic like yours, most of the time it was a problem with the code, not MT5. But the latter happened too.

 

A standard EA like the MACD proved to work properly, i.e. single backtests always matched what was shown in the table of optimization results.

Taking input parameters from my EA manually over from the optimization table to the inputs tab, still does not lead to the displayed result from the optimizations.


First I thought the error "MQL5 start time changed to provide data" might be the cause for the mismatches. However, during tests last night I saw that they also occur when I let the optimizations start at that date where MT5 would change it to.

So, yes, the error must probably be in the code and not done by MT5. Gotta think about how to disable parts of my trading system in order to see when it happens..

 

Could this confusion come from the fact that my EA uses a trendline indicator that computes every tick and running the full optimization cannot be run with real ticks?

The journal lists an error (or warning): "2020.06.19 12:37:34.267 Tester real ticks optimization not allowed in Cloud Network"

 
Marcel Fitzner:

Could this confusion come from the fact that my EA uses a trendline indicator that computes every tick and running the full optimization cannot be run with real ticks?

The journal lists an error (or warning): "2020.06.19 12:37:34.267 Tester real ticks optimization not allowed in Cloud Network"

You are using the Cloud ? You didn't say that previously, you talked about your computer so I thought it was local optimization.
 

Sorry, missed checking back here. Well, I use my LAN with up to 3 computers to do the optimization.

However, the errors occur both when optimizing on a single machine and in the LAN cloud.

Spending tons of time on this, is incredibly frustrating.

I can't rely on the results for 100% (often the results are way off) and codewise I haven't done anything spectacular or uncommon. Simple buy and sell positions in an EA that is getting all signals from an indicator that works fine. No erroneous behaviour during a debugging session - all data is coming through perfectly and on the point.

Maybe I have coded something I'm not aware of that has such an effect on optimization runs, but on the other hand I'm extremely thorough in my coding (extreme clean code, modular approach whenever possible, no external libraries used, stress tests on each utility class, indicator, connector and the EA itself).


Looks like I need to hire someone to show my complete code for an analysis - which to be honest, reveals my whole trading process that I have been working on with a partner for years... Feel like I'm kinda trapped here and close to regret I chose MetaQuotes for a huge trading project.. *sigh

 

Just found something odd:

if I rerun a backtest with the exact same parameters over and over again (here with visualization on),

there's a good chance (I'd say 10%-20%) that something goes really wrong and then instead of being the nice good trade bot...

MCC-Tradebot_working_fine

... it suddenly becomes the weird tradebot, that has super tight SL and TPs:

MCC-Tradebot_erronoues_with_the_same_parameters


Interestingly enough, after I have recompiled the EA and the required Indicator, it seems to have stopped.

Even after restarting a backtest for like 30 times in a row, the weird behavior has stopped.

However, I made a copy of the produced Journal logs, so in case you are interested to see the differences:

So here's the begin of the logs of both runs:

A) Erroneous run:

login (build 2485)
account info found with currency EUR
template file MarginCallCalculator.tpl added. 7286 bytes loaded
expert file added: Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5. 1814890 bytes loaded
program file added: Indicators\Trendline\TrendlineIndicatorPlus.ex5. 361755 bytes loaded
program file added: Indicators\Examples\RSI.ex5. 21825 bytes loaded
program file added: Indicators\Examples\ATR.ex5. 20063 bytes loaded
initial deposit 10000.00 EUR, leverage 1:30
successfully initialized
2155 Kb of total initialization data received
AMD Ryzen 9 3900X 12-Core Processor, 32696 MB
Usa500: symbol to be synchronized
Usa500: symbol synchronized already, 18 bytes received
Usa500: history synchronization started
Usa500: load 27 bytes of history data to synchronize in 0:00:00.000
Usa500: history synchronized from 2017.04.03 to 2020.07.02
Usa500,Daily: history cache allocated for 848 bars and contains 101 bars from 2017.04.03 00:00 to 2017.08.21 00:00
Usa500,Daily: history begins from 2017.04.03 00:00
Usa500,Daily (ActivTrades-Server): 1 minutes OHLC ticks generating
Usa500,Daily: testing of Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5 from 2017.08.22 00:00 to 2020.07.03 00:00 started with inputs: ...

B) Working run:

login (build 2485)
template file MarginCallCalculator.tpl added. 7286 bytes loaded
1482 bytes of tester parameters loaded
85692 bytes of input parameters loaded
5440 bytes of symbols list loaded (853 symbols)
expert file added: Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5. 2259800 bytes loaded
program file added: Indicators\Trendline\TrendlineIndicatorPlus.ex5. 362195 bytes loaded
program file added: Indicators\Examples\RSI.ex5. 21825 bytes loaded
program file added: Indicators\Examples\ATR.ex5. 20063 bytes loaded
17309 Mb available, 216 blocks set for ticks generating
initial deposit 10000.00 EUR, leverage 1:30
successfully initialized
2607 Kb of total initialization data received
AMD Ryzen 9 3900X 12-Core Processor, 32696 MB
Usa500: symbol to be synchronized
Usa500: symbol synchronized, 4040 bytes of symbol info received
Usa500: history synchronization started
Usa500: load 27 bytes of history data to synchronize in 0:00:00.000
Usa500: history synchronized from 2017.04.03 to 2020.07.02
Usa500: start time changed to 2017.08.22 00:00 to provide data at beginning
Usa500,Daily: history cache allocated for 848 bars and contains 101 bars from 2017.04.03 00:00 to 2017.08.21 00:00
Usa500,Daily: history begins from 2017.04.03 00:00
Usa500,Daily (ActivTrades-Server): 1 minutes OHLC ticks generating
Usa500,Daily: testing of Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5 from 2017.04.06 00:00 to 2020.07.03 00:00 started with inputs: ...

The input parameters are the exact same, as you can find it in the attached log files.

Another interesting difference, that is logged later on, is the following:

A) Erroneous run logs 'Execution mode: ENUM_SYMBOL_TRADE_EXECUTION::14538240'
B) In the working run it logs: Execution mode: SYMBOL_TRADE_EXECUTION_REQUEST

So, I'm really curious why there are such differences with the account info and the execution mode.

Any ideas are again very welcome. And also, thank you for your time and help. Really, it'd be much appreciated!

 

Update: I noticed a symbol (EURUSD) that my EA uses for currency conversion is not synchronizing in time when the EA is already past its initialization routine.

I added some proper history loading calls and additional checks found here: https://www.mql5.com/en/forum/265712

"Closing" this thread as all results are now correct.

Synchronize Server Data with Terminal Data
Synchronize Server Data with Terminal Data
  • 2018.07.17
  • www.mql5.com
Is there a way to force the synchronisation process between the Server and Terminal...
 
Alain Verleyen:
You are using the Cloud ? You didn't say that previously, you talked about your computer so I thought it was local optimization.

Hi Alain,


I had a similar issue, but realized that clearing cache finally caused the problem to go away.

Specifically, 2 areas of cache had to be cleared.

1. MetaQuotes\Terminal\<Terminal_ID>\Tester\cache

2. MetaQuotes\Tester\ <Terminal_ID>


Would you be able to point me to an article on how to ensure that the cache do not corrupt? Or articles related to pre-checks that i need to do before an optimization run.

What worries me is that the user would not know when the cache is corrupt.  And we might continue on our optimization doing the wrong thing.

I would like to go about doing optimization without worrying if its corrupt.

If the procedure is to always delete cache before doing a run, i suppose i have to accept it, but it kind of defeats the purpose for it to be called "cache". 

With local at least we can control.

If we are using cloud, how would we know the results are accurate because we will not know if the cache used may be corrupt as well.  Not very good for paying customers rite?




Regards