Systems that perform well in backtest
- EA works in backtest but not in reality
- You're using a "lock"
- Looking for opinions and recommendations
Beautiful topic .
I think that eventhough machine learning has been on the receiving end of allegations about overfitting ,the tests done manually or even forward testing are also prone to it.
If you observe the mechanisms the "trader who prefers to trust his own judgement rather than the machine" utilizes to form or improve upon a strategy are prone to overfitting as well , and even worse , cannot account for the vast amounts of data the machine can.
When i have a strategy input panel and i run it manually test by test in visual mode spending hours away from my girlfriend (or boyfriend) what i'm doing is essentially and clearly without a doubt overfitting.
The first step can be borrowed from machine learning itself . Remove the outliers and flatten everything (normalize) , and balance the outcome rewards based on frequencies of outcomes(or feed samples -in NNs- based on frequencies of outcomes).
And i also believe the first step is testing in the past , can't avoid it.
Edit : So a neural net that overfits essentially is telling me that there is something wrong with the strategy OR with the approach .It's telling me that even if i do it the hardboiled way i will overfit .It will let me know faster and with a deeper certainty because it 100% has seen more data than me.
Edit2 (sorry this coffee is supreme) : What i'm striving for (and failing at) is to make models generalize but i always manage to get them locked in a constant "tug" to one or the other side leaving them mediocre . You probably have noticed this too in your tests.The model that no matter how you use it it fails . Even if you reverse it
Beautiful topic .
I think that eventhough machine learning has been on the receiving end of allegations about overfitting ,the tests done manually or even forward testing are also prone to it.
If you observe the mechanisms the "trader who prefers to trust his own judgement rather than the machine" utilizes to form or improve upon a strategy are prone to overfitting as well , and even worse , cannot account for the vast amounts of data the machine can.
When i have a strategy input panel and i run it manually test by test in visual mode spending hours away from my girlfriend (or boyfriend) what i'm doing is essentially and clearly without a doubt overfitting.
The first step can be borrowed from machine learning itself . Remove the outliers and flatten everything (normalize) , and balance the outcome rewards based on frequencies of outcomes(or feed samples -in NNs- based on frequencies of outcomes).
And i also believe the first step is testing in the past , can't avoid it.
Edit : So a neural net that overfits essentially is telling me that there is something wrong with the strategy OR with the approach .It's telling me that even if i do it the hardboiled way i will overfit .It will let me know faster and with a deeper certainty because it 100% has seen more data than me.
Edit2 (sorry this coffee is supreme) : What i'm striving for (and failing at) is to make models generalize but i always manage to get them locked in a constant "tug" to one or the other side leaving them mediocre . You probably have noticed this too in your tests.The model that no matter how you use it it fails . Even if you reverse it
Very interesting as always Lorentzos. However, it is more related to knowing the type of systems that show up well in backtest, but that in reality does not work, or may not work.
ow i see . sorry . It was a bit ambiguous as many types of systems suffer from this .
For the systems you mentioned , first guess is one is not testing properly.Very interesting as always Lorentzos. However, it is more related to knowing the type of systems that show up well in backtest, but that in reality does not work, or may not work.
If we keep apart over-fitting, there are still a lot of technical traps while using the Strategy Tester.
Firstly the swaps are not historical. So if your strategy is not intra-day you will have more or less serious differences.
Secondly the "trading sessions time" are not historical, so it could also bring some differences if the strategy is sensible to the trading time.
You can also have serious difference if you test on one broker and go live on an other one.
If your strategy is multi-symbols, then the chance you have wrong test results is high unless you know exactly how to avoid the problems (and firstly you need to be aware of the problems).
If you are using something else than real ticks mode, the chance are great that your backtest results are misleading.
You can also have various leverage conditions not reproduce by the Tester. If your strategy is sensible to slippage, just forget the Tester.
And that's about Forex only.
So I would say all strategies are concerned, except checked and proved otherwise. Of course it all depends what you mean with "work" or "doesn't work".
Si dejamos de lado el sobreajuste, todavía hay muchas trampas técnicas al usar el Probador de estrategias.
En primer lugar, los swaps no son históricos. Así que si tu estrategia no es intradiaria tendrás diferencias más o menos graves.
En segundo lugar, el "tiempo de las sesiones de negociación" no es histórico, por lo que también podría traer algunas diferencias si la estrategia es sensible al momento de la negociación.
También puede haber una gran diferencia si realiza la prueba con un corredor y comienza a operar con otro.
Si su estrategia es de múltiples símbolos, entonces la posibilidad de obtener resultados de prueba incorrectos es alta a menos que sepa exactamente cómo evitar los problemas (y primero debe estar consciente de los problemas).
Si está utilizando algo más que el modo de ticks real, es muy probable que los resultados de su prueba retrospectiva sean engañosos.
También puede tener varias condiciones de apalancamiento que el Probador no reproduce. Si su estrategia es sensible al deslizamiento, simplemente olvídese del Probador.
Y eso es sólo sobre Forex.
Por eso yo diría que todas las estrategias están implicadas, salvo que se compruebe y se demuestre lo contrario. Por supuesto, todo depende de lo que quieras decir con "funciona" o "no funciona".
Good point. I think most of the considerations are covered here. There are therefore many weak points in the strategy tester.
Perhaps it would be interesting to work more on this part of Metatrader, including more options such as swaps, commissions, ...
but in the end it seems that the data stored by the brokers is not of the best quality either (intentionally?).
This has been a good starting point, at least for me. Thank you very much for your input!
Good point. I think most of the considerations are covered here. There are therefore many weak points in the strategy tester.
Perhaps it would be interesting to work more on this part of Metatrader, including more options such as swaps, commissions, ...
but in the end it seems that the data stored by the brokers is not of the best quality either (intentionally?).
This has been a good starting point, at least for me. Thank you very much for your input!
Well the customers are the brokers here so we have to factor that in . Why would the broker need historical swaps ?
What i mean to say is there is no pressure (financially) towards these improvements
Good point. I think most of the considerations are covered here. There are therefore many weak points in the strategy tester.
Perhaps it would be interesting to work more on this part of Metatrader, including more options such as swaps, commissions, ...
but in the end it seems that the data stored by the brokers is not of the best quality either (intentionally?).
This has been a good starting point, at least for me. Thank you very much for your input!
What is the "quality" of data ?
1. Very hard to judge. It has nothing to do with strategy but a basic idea could be that symbol is specific and limited to a particular symbol e.g. Gold. User can create
if(_Symbol=="XAUUSD") { double buy_price_01_01_2023 = 1850.00; double sell_price_01_01_2023 = 1890.00; //....... }
a user can create database and this would make EA buy and sell at particular price based on date. It will show fantastic results on EA when backtesting. Definitely he would add some failed trades too to avoid problems, however it reduces chances of scam when a seller publishes LIVE signals of 1 year or more, less than 1 year still means "sum ting wong" which is a reference of some thing wrong in chinese. It is not easy to detect strategy hardcoded in EA until you have source code
2. Ofcourse Latency, Slippage, Human/System error such as network or VPS failure may bring change to results. Commission can be pre-calculated in strategy, good author always add this option inbuilt in EA.
Conclusion : It is very tough to code any working EA which never lose or works above 90% so leverage cant be used and even if any EA has accuracy above 60% it would be only good for leverage free trading, Some scalping strategies i regularly trade are very tough to be coded as they change their character (sl tp) like amoeba every time any price action activity happens on structure.
Reasons can be: overfitting, wrong trade size and risk per trade applied and high spread or commissions compared to the time frame traded.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use