extern variables and timeframe change? - page 2
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Couple of further points prompted by fbj's response:
- My EAs avoid all recovery file reading and writing during strategy test runs.
- Amazingly none of my EAs has ever had a single file read or write malfunction.
- One of the reasons I don't bother matching trades between a recovery file and the trades list is that during init() some of my EAs read the trades list in order that they can "adopt" management of orders that may have been placed manually or "adapt" to the fact that an order under management may have been manually closed. Not saying this is right or wrong. Just how I do it fwiw.
CB
My position is very similar to CB's: most of the EAs I deal with need "state" information above and beyond the raw list of orders in MT4. The nature of this state information is broadly very similar to CB's. The only real difference is that I do tend to store information such as the list of trades which the EA thinks it is managing, partially duplicating the information in MT4's trade list, and then have the EA complain very loudly if there's a mismatch with MT4's trade list when the EA restarts. This has never happened in real life, but I'm a very paranoid person.
Well, that is a definite handling of inconsistency and why not? as money on table and your EA has been kind enough to shout - my current one man band code 'notices' in deinit() any EA owned trades and closes/deletes. Vharsh but "live to fight another day as always another trade around the corner". My kill all is a cop out and the info getting in this thread will push me into being more tolerant - after all, once all datums identified as reqd state mem then easy to do the file bit. Is the data to dump decision making process that is important.
Overall seems fair enough but if real life becomes fraught with many down times, this surely will be user unfriendly? ie, on first thought, seems apparent that reinstated 'old' state compared to a restarted CT will inevitably throw exceptions and as such, manual intervention will be too late if down time window sees trade history diverge significantly... However, nothing wrong with a healthy overdose of paranoia :)
The data simply gets dumped into the experts\files directory, albeit using Win32 file functions rather than the MT4 built-in ones.
I know about [and have:] your public code on this but do not grasp why use if ..\files dir. Reasons? unless is simply case of enhanced functionality and more flexibility if decide to use other HDD areas.
As with CB, there is validation that the state file is recent and not corrupt.
Recent as in embeded timestamp?
Not corrupt as in say simple crc?
The frequency of the storage is user-configurable. All the EAs I deal with are not remotely time-sensitive, and there would be no issue in having them save their state data to disk on every single tick given that the operation typically takes about 4 to 5 milliseconds - the EA would miss a few ticks per minute because it was busy saving state data from the previous tick, but this wouldn't matter.
Missing a tick - yet how does one actually know that? I reckon your approach less hassle and not worth the paranoia. However, against that I'd say that ticks are important if trailing stop (virtual or actual) employed.
The state data then gets continuously copied to multiple remote sites for disaster-recovery purposes. The remote sites hold an archive of the state data rather than just the most recent file, in case corrupt data gets transferred. And, by extremely careful use of order comments, bearing in mind brokers' ability to alter them, I tend to get EAs to record as much of the state information as possible in the MT4 order comment as well. Where possible, this gives them the ability to be started in a special "recovery mode" in the absence of a state file, telling the EA to do the best job it can of recovering its state from the MT4 order history alone. Such a recovery then typically requires manual intervention sooner rather than later to fill in gaps in the information which has been recovered, but it's better than having the EA refuse to do anything at all because its state data is missing or corrupt.
Heavy... appreciate thinking but I'll stick to my nuke 'em all policy - well, until get simple state mem dump/restore going albeit not at all dynamic like you scheme. Yet your methods are worth investigating when the time allows.
Well, this is all just great grist to 'da mill and seriously appreciate your time taken to respond.
Thanks
Overall seems fair enough but if real life becomes fraught with many down times, this surely will be user unfriendly? ie, on first thought, seems apparent that reinstated 'old' state compared to a restarted CT will inevitably throw exceptions and as such, manual intervention will be too late if down time window sees trade history diverge significantly... However, nothing wrong with a healthy overdose of paranoia :)
It's working on the basis that, if there were a discrepancy between what it was expecting versus what MT4 reported on restart, it wouldn't be safe to continue automated trading. It would imply a big problem either with MT4 or with the broker's systems.
I know about [and have:] your public code on this but do not grasp why use if ..\files dir. Reasons? unless is simply case of enhanced functionality and more flexibility if decide to use other HDD areas.
It uses the experts\files directory simply because I started out using the built-in MT4 functions. I switched to Win32 file functions because of the ability to open files with shared access. Without this, I was getting contention issues between the EAs writing files and the replication processes trying to read them. The other reason for using experts\files rather than e.g. Application Data is that it means that you can still do a complete copy of an MT4 instance and all the data it needs simply by copying the root directory.
Missing a tick - yet how does one actually know that?
I don't know, but it's not unlikely that you'll get a couple of ticks within milliseconds of each other during a busy market. (But, as I said, the frequency of the writes is user-configurable.)
Recent as in embeded timestamp?
Not corrupt as in say simple crc?
I'm actually not too fussed about "recent". I've got separate external monitoring processes which watch EAs to make sure that they haven't somehow died.
I probably should have a crc in the state files to validate them, but at the moment I'm just simply using a file terminator. If it isn't present, the file is potentially partial and unreliable. (However, corruption in the middle of the file would stand a good chance of causing a mismatch between the state information versus what MT4 is reporting, which takes us back to the beginning again...)
Couple of further points prompted by fbj's response:
- My EAs avoid all recovery file reading and writing during strategy test runs.
yes, navigate around superfluous paths for sure! I'm currently having a devil of a time retro fitting such thunks to my stuff and all of a sudden being forced into deep immersion maintenance mode and [yet again] observing that one's design is in some areas sadly lacking the 'easy to maintain' attribute - oh merde!
Only thing that keeps me laughing is the copious comments I generally leave in code otherwise would not have a clue what's up doc!
- Amazingly none of my EAs has ever had a single file read or write malfunction. That's heartening CB, just goes to show how mtbf's have increased to infinity and beyond...:)
- One of the reasons I don't bother matching trades between a recovery file and the trades list is that during init() some of my EAs read the trades list in order that they can "adopt" management of orders that may have been placed manually or "adapt" to the fact that an order under management may have been manually closed. Not saying this is right or wrong. Just how I do it fwiw.
This is interesting idea. Very proactive. Other than sym,per how do you ident manually placed trades as being 'adoptable' - unless rely also on comment ?
jjc has a complex recovery design which I surely appreciate but duplicating MT data certainly can increase complexity. I used to have trade state mem until read one of phy's posts... On the other hand, some 'per trade' data is not findable unless remembered so yes... another candidate for dump file...
CBMore goodies to file and digest into workable practices for my setup.
Thanks
It's working on the basis that, if there were a discrepancy between what it was expecting versus what MT4 reported on restart, it wouldn't be safe to continue automated trading. It would imply a big problem either with MT4 or with the broker's systems.
Am getting more into your idea. A keyword that woke me up was automated and to be frank that 'is' name of game after all. ie, automated winning NOT loosing!
It uses the experts\files directory simply because I started out using the built-in MT4 functions. I switched to Win32 file functions because of the ability to open files with shared access. Without this, I was getting contention issues between the EAs writing files and the replication processes trying to read them. The other reason for using experts\files rather than e.g. Application Data is that it means that you can still do a complete copy of an MT4 instance and all the data it needs simply by copying the root directory.
I don't know, but it's not unlikely that you'll get a couple of ticks within milliseconds of each other during a busy market. (But, as I said, the frequency of the writes is user-configurable.)
The ability to replicate by install tree copy is good feature. I'd make one proviso on that ease of duplication in that it is also very easy to duplicate one's coding efforts unless strict rules made that only one instance is ever used for coding etc.
Source code reconcilliation is not fun if have >1 same named file and each has adhoc additions
jjc
Your restart recovery uses saved states.
What about during trading and maintaining in sync with MT?
ie, orders can close at any time.
What strategy imployed to maintain internal DB sychronisation with MT ?
And is this prioity action done each entry?
fbj wrote >>
The ability to replicate by install tree copy is good feature. I'd make one proviso on that ease of duplication in that it is also very easy to duplicate one's coding efforts unless strict rules made that only one instance is ever used for coding etc.
Source code reconcilliation is not fun if have >1 same named file and each has adhoc additions
The way I personally handle this is to have the mq4 file outside of the MT4 instances in a completely separate area of disk. I then use a batch file which compiles the mq4 using an old build of metalang.exe while tweaking the computer's clock, and which copies the resulting ex4 file into the MT4 instances.
fbj wrote >>
I used to have trade state mem until read one of phy's posts...
Wherever possible I do avoid having to maintain trade state information. However, more often than not I'm dealing with EAs which need information about trades beyond the raw prices etc reported by MT4. For example, they need to know why a trade was placed. In addition, individual trades are often part of a wider context, and the EA needs to know how an individual trade fits into the overall strategy. In some cases the EA could reconstruct this information by looking at historic data, but that would be (substantially) more difficult than keeping track of state data.
fbj wrote >>
Your restart recovery uses saved states.
What about during trading and maintaining in sync with MT?
ie, orders can close at any time.
What strategy imployed to maintain internal DB sychronisation with MT ?
And is this prioity action done each entry?
The saved state is only read in init(). Thereafter, state information is held in memory, and the state file on disk is simply repeatedly overwritten. In most cases, my EAs scan the MT4 order list and update their internal state on every tick. I don't have any EAs which "argue" with MT4 - if MT4 says that an order is filled, or closed, or deleted, then the EA just accepts that as being the case. None of my EAs have a need to ask themselves "Should this have happened?"
The way I personally handle this is to have the mq4 file outside of the MT4 instances in a completely separate area of disk. I then use a batch file which compiles the mq4 using an old build of metalang.exe while tweaking the computer's clock, and which copies the resulting ex4 file into the MT4 instances.
Wherever possible I do avoid having to maintain trade state information. However, more often than not I'm dealing with EAs which need information about trades beyond the raw prices etc reported by MT4. For example, they need to know why a trade was placed. In addition, individual trades are often part of a wider context, and the EA needs to know how an individual trade fits into the overall strategy. In some cases the EA could reconstruct this information by looking at historic data, but that would be (substantially) more difficult than keeping track of state data.
The saved state is only read in init(). Thereafter, state information is held in memory, and the state file on disk is simply repeatedly overwritten. In most cases, my EAs scan the MT4 order list and update their internal state on every tick. I don't have any EAs which "argue" with MT4 - if MT4 says that an order is filled, or closed, or deleted, then the EA just accepts that as being the case. None of my EAs have a need to ask themselves "Should this have happened?"
Firstly, thank you for your inputs, they help to clarify thinking :)
And
bit late in day but...
I have, in the past used the following snippet at init().top
if(iCode==REASON_CHARTCHANGE || iCode==REASON_PARAMETERS) {...eg:inhibit resetting certain EA globals and or?...}
Reasons:
1. to inhibit automake of new EA.ID, which I use along with ascending sequential order#, to key each new trades magic#
2. in multi EA context, the forgetting/resetting of one's EA.ID, especially if open orders at time of chartChange or param reasons would stop EA being able to map its owned orders via magic#...
3. Global and static memory state retained - as observed from tests.
NOTE:I do remain sceptical regards this whole uninit.. ClientTerminal design and have to admit to getting inconsistent test results in the past over just what memory is persistent relative to reason codes... Basically, I would rather take the KISS/Draconian attitude that should init() and/or deinit() be called [by my code or interpreter] that the EA behaves asif cleansheet in init() and shutdown in deinit(). Perhaps someone can help here too. But, (2) above appears to be an issue that requires taking notice of uninit.. codes. Maybe other ways are available to sidestep (2) issues and not have to deal with uninit.. codes?
.
I would be very keen to know how you and others willing to post,
1. deal with the various uninit.. reason codes (please include "reasons" for doing)
2. Tackle 3.NOTE
.
Thanks in advance
Basically, I would rather take the KISS/Draconian attitude that should init() and/or deinit() be called [by my code or interpreter] that the EA behaves asif cleansheet in init() and shutdown in deinit(). Perhaps someone can help here too.
I'm in the happy position that I more or less don't have to worry about the same EA being used on multiple charts for the same symbol. My code does typically cater for this, but only via the cumbersome, sub-optimal route of requiring the user to manually assign a unique ID to each EA via an extern property. But no-one's ever actually used this in live trading.
Starting from this position, my EAs don't need to care why they're being unloaded or reloaded. They just save everything (in deinit(), and also on individual ticks), and don't bother to check in init() whether any data has been retained in memory. They simply reload everything from disk, overwriting any values which have persisted in memory. If no state file exists on disk when init() is called, then the EA assumes that it's starting up for the first time.
I'm in the happy position that I more or less don't have to worry about the same EA being used on multiple charts for the same symbol. My code does typically cater for this, but only via the cumbersome, sub-optimal route of requiring the user to manually assign a unique ID to each EA via an extern property. But no-one's ever actually used this in live trading.
Starting from this position, my EAs don't need to care why they're being unloaded or reloaded. They just save everything (in deinit(), and also on individual ticks), and don't bother to check in init() whether any data has been retained in memory. They simply reload everything from disk, overwriting any values which have persisted in memory. If no state file exists on disk when init() is called, then the EA assumes that it's starting up for the first time.
Well jjc, thanks again. My todo list now has the various functionalities you and CB have been kind to share. I'll admit to having generally cruised around the periphery for many years but never got the shovel out :/
It would allow a cleaner and simpler structure to take shape. Also, I like the idea of taking charge of one's data and not relying on CT and it can be black boxed such that any code can use. Well, with mods as reqd... oh for ptrs and templates and structs. But I'll not be using '5' - will stick with '4' until forced to migrate by Broker. Ha haaa, by that time all the bleeding edge types will have winkled out many of the inevitable gremlins which are going to surface. But given 4's fix hist, this is gonna be years in the doing...
Regards