Strategy Testing Bug

[Travis Silvers]

While working with the Strategy tester to optimize an experts parameters I have run into an issue where when optimizing the Balance + Profit factor the result ends up blank.

I export to XML and see that the result appears to be the max size for a double.

I'm not sure what has caused this but it is repeatable. 

Anyone else experiencing this?

Build 478 

[Slava]

DBL_MAX assumed as empty value

[Travis Silvers]

Curious. Does that affect the GA during optimization to have a result with a value set at DBL_MAX? I hope that doesn't skew the optimization towards those results.

When exported to XML it results in a value with all of the zeros to represent DBL_MAX, which causes a problem when trying to open the XML in Excel. If it were to instead use scientific notation or another value to represent an empty value when written to XML that would prevent Excel from having a problem.

[Slava]

You should know how calculated the profit_factor value

profit_factor = gross_profit / gross_loss;

If gross loss is zero then profit factor cannot be calculated and its value becomes DBL_MAX (NOT ZERO!!!) 

[Travis Silvers]

Thank you for the information. I assumed it was to prevent some type of divide by zero issue or overflow.

I would still like to know how that effects the GA optimization process? Typically the "Result" with the largest value drives the optimization routine, so if the value is DBL_MAX it could potentially and erroneously drive optimization towards those results. It is of course possible that the optimization routine accounts for this and that is the information I am looking for.

Thanks 

[Slava]

firestrand:

I would still like to know how that effects the GA optimization process? Typically the "Result" with the largest value drives the optimization routine, so if the value is DBL_MAX it could potentially and erroneously drive optimization towards those results. It is of course possible that the optimization routine accounts for this and that is the information I am looking for.

DBL_MAX is  the best result for GA. And there is no error because gross loss is 0 - You lost nothing

[Travis Silvers]

stringo:
DBL_MAX is  the best result for GA. And there is no error because gross loss is 0 - You lost nothing

From an optimization standpoint I have to disagree. The optimization is supposed to be for max profit factor and balance, so when the gross loss has a 0 value, the profit factor is apparently set to DBL_MAX as a way to prevent a divide by zero. This appears to happen when a parameter set results in a small number of trades, however the actual amount of profit and final balance aren't necessarily inline with the best performing set of parameters. It is especially troubling when forward testing, as the forward testing routine typically takes a sample of the "top" results, which aren't necessarily the best due to the huge skew from the DBL_MAX profit factor. There are many possible ways to fix this but for now using profit factor with balance and forward testing is crippled from my viewpoint. I'll look into "Custom" and see if there is anything I can do there.

[Slava]

Please more details about "many possible ways"

I see only one way. If gross profit is 0 (ie 0 trades because gross loss is 0 too) profit factor may be assumed as 0. Else profit factor will be DBL_MAX.

[Travis Silvers]

stringo:

Please more details about "many possible ways"

I see only one way. If gross profit is 0 (ie 0 trades because gross loss is 0 too) profit factor may be assumed as 0. Else profit factor will be DBL_MAX.

Sure, here are a couple ideas off the top of my head.

1. Replace the 0 protected division and instead of returning DBL_MAX when gross_loss == 0, force gross_loss to a minimum value of 1. Though this would cause profit_factor to equal gross_profit which doesn't sound right.

2. Replace the DBL_MAX result when gross_loss == 0 with 10000 or some other non-scale busting result, as in theory profit_factor is supposed to be the "ratio of the gross profit to the gross loss in percents" and typically you don't see any 10000:1 ratios. This could be made dynamic by always making the result larger than any other found, but that seems more complex.

3. Use a squashing function such as the sigmoid** function on the results of both parameters being optimized. In my case this would force both profit factor and balance to range from 0-1. This would prevent the DBL_MAX when gross_loss = 0 to overpower all other results.

I would recommend #3 which is exactly what I plan to do when I figure out the custom option. 

Empirical analysis so far has shown that the DBL_MAX result tends to skew the results of the GA optimization towards results with 1 or 2 trades with 0 loss. Even on M1 and M5 periodicity. However when I optimize by balance only the results are much more in line with a potentially profitable system. 

 **Sigmoid or Logistic function P(t) = 1 / (1 + e^(-t)) where e is the natural number and t is the value being squashed.

[Travis Silvers]

Here is what I have implemented in my EA for the "Custom" fitness:

double OnTester()
{
   double result = 0.0;
   result += Sigmoid(TesterStatistics(STAT_PROFIT_FACTOR));
   result += Sigmoid(TesterStatistics(STAT_SHARPE_RATIO));
   result += Sigmoid(TesterStatistics(STAT_PROFIT));
   return result * 100.0;
}

double Sigmoid(double value)
{
   return 1 / ( 1 + MathPow(M_E, -value));
}

 Enjoy!

[Slava]

We will think about it

Thank you