When to program an EA for Automated Trading?
1. Should I manually or visual back test different trading strategies by hand for profitability before programming or paying for an EA? I think the benefit of this is not spending so much time programming a strategy that is not profitable, but I do get programming experience. Also, it gives me a chance to visualize all the what-if scenarios or if the strategy makes sense to me.
2. Should I just program different trading strategies that fit my liking and back test for profitability? I believe this will be faster then manual back testing and build programming experience. But will take some time.
3. If hire a programmer to build an EA, should I be concern the programmer will steal my EA for his/her own trading since programmer can back test my EA as well and see the results. How do I trust the privacy of my paid EA and protect my trading system?
- It is my strong opinion, that you should always manually back-test a strategy even before considering constructing an EA for it. If one is a very good programmer and feel tempted to just code it directly, one should resist such an urge. Doing the back-test manually allows us as traders to "visually" experience some of the nuances that may crop up and to take note of them. Doing it manually will allow us to think of the different situations and how to approach them. Such situations may not be easily identified if just blindly running an EA and only looking at the end results. The EA could in fact have unidentified bugs that could skew the results and since one did not carry out a manual back-test, it would go unnoticed and incorrectly cause one to make a decision on its profitability based on a false premise.
- No! It may seem faster, but will be a waste of time because you will just be "grabbing at straws", blindly hacking at ideas and variations without really contemplating the effectiveness of those ideas. It is much more productive to actually ponder and consider each idea and variation properly and thoroughly, considering mathematical and statistical implications as well as manually back-test them to see how they play out and what changes may be necessary.
- Yes, obviously if your strategy is a profitable one, the programmer will obviously make use of it too, if so inclined.
- In fact, if the coder is good at his skill, I would even encourage you to be forthcoming and shared some of the settings for the parameters, so that the coder will be motivated to do a more thorough job and actually feel interested in the project. He may even contribute with extra features due to his own enthusiasm and offer better support after the fact, such as working on bugs and offer improvements as time goes by. He may even offer to work on other projects for you for free, if your EA turns out to be good and profitable.
- If however, you want to keep it completely private and not fear that it might be used by others, then you have only one alternative, and that is to code it yourself. Even if you consider a non-disclosure agreement with the coder, the fact is that it actually has no real effect, as the coder can easily still make his own variation, use it or even sell it to others. The only guaranteed way to never have it disseminated, is to code it yourself.
- However, please note, that the vast majority of strategies in EA form are NOT profitable. Many still need human intervention to decide when to use them or not, or require constant adjustment of parameters in consideration of market conditions. Only a very, very small percentage of EAs can actually be profitable consistently with little human intervention, and these can only be possible with the following conditions:
- Good strategies backed by mathematical and statistical principals,
- Excellent Risk Assessment and Money Management techniques,
- Well designed and structured code carried out by a programmer with very good coding skills. However, such programmers are difficult to find and are also quite expensive.
- It is my strong opinion, that you should always manually back-test a strategy even before considering constructing an EA for it. If one is a very good programmer and feel tempted to just code it directly, one should resist such an urge. Doing the back-test manually allows us as traders to "visually" experience some of the nuances that may crop up and to take note of them. Doing it manually will allow us to think of the different situations and how to approach them. Such situations may not be easily identified if just blindly running an EA and only looking at the end results. The EA could in fact have unidentified bugs that could skew the results and since one did not carry out a manual back-test, it would go unnoticed and incorrectly cause one to make a decision on its profitability based on a false premise.
- No! It may seem faster, but will be a waste of time because you will just be "grabbing at straws", blindly hacking at ideas and variations without really contemplating the effectiveness of those ideas. It is much more productive to actually ponder and consider each idea and variation properly and thoroughly, considering mathematical and statistical implications as well as manually back-test them to see how they play out and what changes may be necessary.
- Yes, obviously if your strategy is a profitable one, the programmer will obviously make use of it too, if so inclined.
- In fact, if the coder is good at his skill, I would even encourage you to be forthcoming and shared some of the settings for the parameters, so that the coder will be motivated to do a more thorough job and actually feel interested in the project. He may even contribute with extra features due to his own enthusiasm and offer better support after the fact, such as working on bugs and offer improvements as time goes by. He may even offer to work on other projects for you for free, if your EA turns out to be good and profitable.
- If however, you want to keep it completely private and not fear that it might be used by others, then you have only one alternative, and that is to code it yourself. Even if you consider a non-disclosure agreement with the coder, the fact is that it actually has no real effect, as the coder can easily still make his own variation, use it or even sell it to others. The only guaranteed way to never have it disseminated, is to code it yourself.
- However, please note, that the vast majority of strategies in EA form are NOT profitable. Many still need human intervention to decide when to use them or not, or require constant adjustment of parameters in consideration of market conditions. Only a very, very small percentage of EAs can actually be profitable consistently with little human intervention, and these can only be possible with the following conditions:
- Good strategies backed by mathematical and statistical principals,
- Excellent Risk Assessment and Money Management techniques,
- Well designed and structured code carried out by a programmer with very good coding skills. However, such programmers are difficult to find and are also quite expensive.
Thank you so much Fernando Carreiro for your simple and effective response to my message. Really appreciate it.
Sometimes its a bit confusing knowing what to do when looking for a strategy to back test and eventually automate.
Everything you mentioned make sense to me, manual back testing allows the developer/trader to become intimate with the strategy details/mechanics. I agree with you about working closely with the programmer and sharing profitable ideas with him so he/she is energetic about the EA projects.
Questions:
3. What is your recommended resource (book, website, brainstorming,etc.) for find stratgies to consider for an EA? Lately, I just browse the internet for strategies that I like, which or trending swing trades (on 1 hour to 15 min chart).
4. May I please have an example of what you mean by "Good strategies backed by mathematical and statistical principals"? Are you referring to a back test performance metrics?
Thank you kindly
Thank you so much Fernando Carreiro for your simple and effective response to my message. Really appreciate it.
Sometimes its a bit confusing knowing what to do when looking for a strategy to back test and eventually automate.
Everything you mentioned make sense to me, manual back testing allows the developer/trader to become intimate with the strategy details/mechanics. I agree with you about working closely with the programmer and sharing profitable ideas with him so he/she is energetic about the EA projects.
Questions:
3. What is your recommended resource (book, website, brainstorming,etc.) for find stratgies to consider for an EA? Lately, I just browse the internet for strategies that I like, which or trending swing trades (on 1 hour to 15 min chart).
4. May I please have an example of what you mean by "Good strategies backed by mathematical and statistical principals"? Are you referring to a back test performance metrics?
Thank you kindly
For query 1 & 2:
I break the process up into various stages:
- Stage 1 - Pre-checking:
Before doing anything else, I need to define the basic rules, so I will test out the idea with maybe just 5 trades per chart set-up, on various symbols and time-frames, just to get a feel for the concept and its merits. This quickly weeds out any ideas without solid backing and helps me define the exact rules to be used for the strategy.- Stage 2 - Analysis:
At this stage, I have defined the main rules of the strategy, so I select a single Symbol and Time-frame (based on which seemed the most promising in the stage 1 tests). Then I proceed to test at least 20 trades, that will be recorded on a spreadsheet, including calculations for various metrics, based on a fixed average spread & commission value and a 1% fractional risk (not fixed volume, but actual risk assessed volume). I record and calculate the individual values such as MAE, MFE, R:R, Gain, etc. but also running totals for Profit, Loss, Draw-down, Profit Factor, Win Rate as well as a few graphs for Balance and other running totals. I also take note of observations and comments, possible changes to rules or filters to be used. However, through out the process I do not deviate from the initial rules. I only take note of the possible changes.- Stage 3 - Refinement:
I now consider the possible changes to the rules and code an indicator which will aid in identifying the basic entry and exit setups for the now refined rules. With this basic indicator, I continue to manually test at least another 20 trades and continue to record all aspects in the spreadsheet as well as further observations and comments.- Stage 4 - The devil is in the details:
At this point, I now know exactly how the rules should be so I code a fully functional "smart" indicator, that contains all the rules, and nuances including trailing stops, filters, etc. - all but the actual trading process, as this is is still an Indicator and not an EA. I then proceed to manually test this final version of the strategy rules, with the aid of this sophisticated "smart" indicator, for at least another 60 trades, while still recording each piece of information and tracking the metrics, and still taking note of observations and comments on the trades (or bugs in the indicator).- Stage 5 - Wrapping-it-up:
I now have at least 100 trades manually tested. With the running tab of all my metrics, not only can I see the results of each trade, but I can also plainly see how the balance graph looks like and what the profit factor, win rate and other metrics look like for the entire set. So, I can now make a final judgement on whether this strategy has promise or not.- Stage 6 - Automation:
Now that I have all the details on how to trade the strategy, I can now work on an EA, that will take the "smart" Indicator, and automate the trading process. I will also refine the strategy based on the overall analysis of the results and all the observations and comments recorded. This however, is not a fully functional EA that can be used for live trading, but instead an EA specifically build for advanced back-testing, that will also record and calculate metrics that are not available on MetaTrader.- Stage 7 - Diversified Back-testing:
Now, I will carry out back-testing using the MetaTrader's Strategy Tester (always using real tick data) on various Symbols and Time-frames in order to get a feel for the overall impact it has and to see in a definitive manner how promising it can be in terms of profitability. I will also carry out some optimisation tests if needed, but this is usually rare, as I try to develop strategies that don't use fixed values but instead rely on market variables and indicator values, in order to define sizes for stops (such as S/L, T/S, B/E, T/P) as well as some other indicator parameters.- Stage 8 - Forward Testing:
At this stage I will improve the EA in order for it to safely be able to trade Live and proceed with Live Forward testing. However, I only carry out these forward tests in some cases. At this point all the manual tests, and automated tests have given me more than enough confidence to know whether it will work on Real Live trading or not. I usually only go ahead with a live forward test, when the strategy is highly susceptible to spread variations or other types of erratic behavior or volatility. Other than these situations, I usually skip this stage. So far, in my own experience, it has not been a high priority for me. However, this depends on the type of strategy and one's own trading knowledge, skill and experience. So, it might be a good idea that you don't skip this step in your own testing until you can decide whether it is of high importance or not for your own particular situation.- Stage 9 - Final Review:
And finally I have all I need to decide whether to go forward or not, and if yes I now totally rewrite a new EA, specifically designed for fast, lean and efficient use on live trading without any code "bloat" or visual distractions.PS! For short time-frame strategies, I will usually extend the number of trades manually back-tested, to at least have a full month, full week or a full day depending on how low/short the tested time-frame. For the higher time-frames, if feasible, I try to have a full year of trades.
For query 3:
In the beginning, I would look to books and/or web sources for trading strategies or ideas. Currently, however, while I might still browse those sources, I prefer relying on my own whit, and observations of the charts I see before me, and coming up with ideas of my own. If feels more natural to do it this way and tends to be more effective then trying out other peoples' strategies when one has no idea how they came up with them or if they simply "sucked on their thumb" and out pop some nonsense that just sounded good. At least, when our strategies are our own, we know how much merit may be behind them in order to decide to pursue them or not.
For query 4:
When I come up with ideas (or find them in books or web sources), I no longer just accept them at face value (even my own ideas). I open a sheet on MathCAD and proceed to describe the idea mathematically in order to further analyse the mathematical or statistical principals behind them. If it is more of a statistical basis, I may code a script or EA, to run against Market data to analyse the probabilities behind such a strategy (i.e. the "Edge"). Only once these mathematical or statistical principles have been fully analysed and the results look to be promising, do I advance to Stage 1 of the back-testing.
PS! Instead of MathCAD you could use some other tool such as MatLab (or even a spreadsheet but that does have its limits).
Good stuff Fernando. This could be an article.
Thank you for the complement! However, MetaQuotes would not be interested in such an article (see below).
Forum on trading, automated trading systems and testing trading strategies
Rashid Umarov, 2017.02.03 08:12
Kraisorn Saengkae:What about article about technical analysis which is not necessarily MQL5 programming, do you accept that too ?
We do not want general article without practical approach. Every article must give to traders/readers something usefull, what they can prove in MetaTrader 5.
Code, code and one more code.
For query 1 & 2:
I break the process up into various stages:
- Stage 1 - Pre-checking:
Before doing anything else, I need to define the basic rules, so I will test out the idea with maybe just 5 trades per chart set-up, on various symbols and time-frames, just to get a feel for the concept and its merits. This quickly weeds out any ideas without solid backing and helps me define the exact rules to be used for the strategy.- Stage 2 - Analysis:
At this stage, I have defined the main rules of the strategy, so I select a single Symbol and Time-frame (based on which seemed the most promising in the stage 1 tests). Then I proceed to test at least 20 trades, that will be recorded on a spreadsheet, including calculations for various metrics, based on a fixed average spread & commission value and a 1% fractional risk (not fixed volume, but actual risk assessed volume). I record and calculate the individual values such as MAE, MFE, R:R, Gain, etc. but also running totals for Profit, Loss, Draw-down, Profit Factor, Win Rate as well as a few graphs for Balance and other running totals. I also take note of observations and comments, possible changes to rules or filters to be used. However, through out the process I do not deviate from the initial rules. I only take note of the possible changes.- Stage 3 - Refinement:
I now consider the possible changes to the rules and code an indicator which will aid in identifying the basic entry and exit setups for the now refined rules. With this basic indicator, I continue to manually test at least another 20 trades and continue to record all aspects in the spreadsheet as well as further observations and comments.- Stage 4 - The devil is in the details:
At this point, I now know exactly how the rules should be so I code a fully functional "smart" indicator, that contains all the rules, and nuances including trailing stops, filters, etc. - all but the actual trading process, as this is is still an Indicator and not an EA. I then proceed to manually test this final version of the strategy rules, with the aid of this sophisticated "smart" indicator, for at least another 60 trades, while still recording each piece of information and tracking the metrics, and still taking note of observations and comments on the trades (or bugs in the indicator).- Stage 5 - Wrapping-it-up:
I now have at least 100 trades manually tested. With the running tab of all my metrics, not only can I see the results of each trade, but I can also plainly see how the balance graph looks like and what the profit factor, win rate and other metrics look like for the entire set. So, I can now make a final judgement on whether this strategy has promise or not.- Stage 6 - Automation:
Now that I have all the details on how to trade the strategy, I can now work on an EA, that will take the "smart" Indicator, and automate the trading process. I will also refine the strategy based on the overall analysis of the results and all the observations and comments recorded. This however, is not a fully functional EA that can be used for live trading, but instead an EA specifically build for advanced back-testing, that will also record and calculate metrics that are not available on MetaTrader.- Stage 7 - Diversified Back-testing:
Now, I will carry out back-testing using the MetaTrader's Strategy Tester (always using real tick data) on various Symbols and Time-frames in order to get a feel for the overall impact it has and to see in a definitive manner how promising it can be in terms of profitability. I will also carry out some optimisation tests if needed, but this is usually rare, as I try to develop strategies that don't use fixed values but instead rely on market variables and indicator values, in order to define sizes for stops (such as S/L, T/S, B/E, T/P) as well as some other indicator parameters.- Stage 8 - Forward Testing:
At this stage I will improve the EA in order for it to safely be able to trade Live and proceed with Live Forward testing. However, I only carry out these forward tests in some cases. At this point all the manual tests, and automated tests have given me more than enough confidence to know whether it will work on Real Live trading or not. I usually only go ahead with a live forward test, when the strategy is highly susceptible to spread variations or other types of erratic behavior or volatility. Other than these situations, I usually skip this stage. So far, in my own experience, it has not been a high priority for me. However, this depends on the type of strategy and one's own trading knowledge, skill and experience. So, it might be a good idea that you don't skip this step in your own testing until you can decide whether it is of high importance or not for your own particular situation.- Stage 9 - Final Review:
And finally I have all I need to decide whether to go forward or not, and if yes I now totally rewrite a new EA, specifically designed for fast, lean and efficient use on live trading without any code "bloat" or visual distractions.PS! For short time-frame strategies, I will usually extend the number of trades manually back-tested, to at least have a full month, full week or a full day depending on how low/short the tested time-frame. For the higher time-frames, if feasible, I try to have a full year of trades.
For query 3:
In the beginning, I would look to books and/or web sources for trading strategies or ideas. Currently, however, while I might still browse those sources, I prefer relying on my own whit, and observations of the charts I see before me, and coming up with ideas of my own. If feels more natural to do it this way and tends to be more effective then trying out other peoples' strategies when one has no idea how they came up with them or if they simply "sucked on their thumb" and out pop some nonsense that just sounded good. At least, when our strategies are our own, we know how much merit may be behind them in order to decide to pursue them or not.
For query 4:
When I come up with ideas (or find them in books or web sources), I no longer just accept them at face value (even my own ideas). I open a sheet on MathCAD and proceed to describe the idea mathematically in order to further analyse the mathematical or statistical principals behind them. If it is more of a statistical basis, I may code a script or EA, to run against Market data to analyse the probabilities behind such a strategy (i.e. the "Edge"). Only once these mathematical or statistical principles have been fully analysed and the results look to be promising, do I advance to Stage 1 of the back-testing.
PS! Instead of MathCAD you could use some other tool such as MatLab (or even a spreadsheet but that does have its limits).
Thank you so much Fernando Carreiro for your effort and time in writing a detail response to my questions.
Very interesting. I will need to read the post again and again to understand. What you state make sense cause you trying to determine before coding by hand and stats and sense of edge that the strategy will perform well during a historical back test when programmed. After all, programming strategy does take time.
I like your scientific and analytical approach to finding and hand testing a strategy before moving forward to development stage. I'll need some time to think your stages through. It has been 10 years since I used Matlab in school, lol, but it is for high end mathematically programming.
Thank you
Thank you for the complement! However, MetaQuotes would not be interested in such an article (see below).
Thank you for the complement! However, MetaQuotes would not be interested in such an article (see below).
Its all about technical analysis, that's the only tool must traders use. It is definitely worth an article for readers.
Very interesting. I will need to read the post again and again to understand. What you state make sense cause you trying to determine before coding by hand and stats and sense of edge that the strategy will perform well during a historical back test when programmed. After all, programming strategy does take time.
I like your scientific and analytical approach to finding and hand testing a strategy before moving forward to development stage. I'll need some time to think your stages through. It has been 10 years since I used Matlab in school, lol, but it is for high end mathematically programming.
Thank you
Here is an example of a spreadsheet for one such back-test (attached ZIP archive, containing an Excel ".xlsx" file).
PS! MatLab is obviously more powerful, but I personally prefer using MathCAD because it is much more visual - like a word processor for maths.
Here is an example of a spreadsheet for one such back-test (attached ZIP archive, containing an Excel ".xlsx" file).
PS! MatLab is obviously more powerful, but I personally prefer using MathCAD because it is much more visual - like a word processor for maths.
Thank you very much Fernanodo for the spreadsheet. It is nice put together for a manual back test. I have similar one that calculates positive expectancy for me as well. i still haven't figured out to write a Excel formula to calculate drawdown from back tested trades.
Few more questions please:
1. I am confused on stage 3 and 4. What do you mean by " fully functional "smart" indicator,"? So you no longer need to manually back test by hand, the indicator will trade for you?
2. What do you mean by filter? I see in your spreadsheet and stages above you mention filter.
3. What is your opinion or advice or comments regarding attempting to eventually build an EA for the crytopcurrency market? I personally think due to lack historical data its not good idea. It just started trending this year. Just my opinion.
Thanks
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hello,
I have a few questions as I do have trading manually experience but need to trade with EA automated due to working full time now. I have programming experience in NinjaTrader programming, but NT does not support cryptocurrency data feed market and I would like to trade this market now, and I like MT4. So I have to learn how to program in MT4 now.
1. Should I manually or visual back test different trading strategies by hand for profitability before programming or paying for an EA? I think the benefit of this is not spending so much time programming a strategy that is not profitable, but I do get programming experience. Also, it gives me a chance to visualize all the what-if scenarios or if the strategy makes sense to me.
2. Should I just program different trading strategies that fit my liking and back test for profitability? I believe this will be faster then manual back testing and build programming experience. But will take some time.
3. If hire a programmer to build an EA, should I be concern the programmer will steal my EA for his/her own trading since programmer can back test my EA as well and see the results. How do I trust the privacy of my paid EA and protect my trading system?
Great forum!
Thank you for the help.