Need help! Can't solve the problem, I'm hitting hardware limitations - page 15
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Finally figured out what is needed, hopefully this version is final.
As said Everything is simple enough to split and parallel, the main steps:
1. a. It would be very useful to know where the beginning and the end of each sequence (because of this I suggested last time to write their sizes in a separate file)
б. You can parse them yourself, but in this case you will have to come back when reading the next part of file from disk to read the previous trimmed sequence
Next I will consider variant 1.a. Probably, it's not optimal, but I like it better.
2. Knowing sizes of sequences and memory size for a part of file (500 mb), we can calculate the size of the part of the file we need to download.
3. In parallel, we calculate coefficients for sequences, as we know the beginning and end of each sequence.
4. Beginning and end of each sequence can be stored in the multithreaded queue (filled while calculating step 2)
5. Calculation result - structure (array from structure where time and coefficient + sequence number))
6. When half of the initial size of the allocated memory (250 mb) is processed, the process of rewrite is started with the formation of the second queue with the beginning and the end
7. Having reached the end of the first line we read from the second one; having reached the end of the second line we read from the first one.
8. You can simultaneously calculate coefficients and read from the file.
9. But each result of calculation of coefficients shall be stored, and it is better to merge them at once in the response:
We have to write merging functions: two sequences of coefficients into one subresult, two subresults into one slightly complete subresult
10. The merging process can also be done in parallel.
11. When is the result? When the sizes of sequences read from the add-on file are finished, then two stacks will become empty, then the calculation of coefficients will be finished,
then we have to wait for the merge of subresult process to finish (it can also be done via thread-safe queue)
and finally merge subresult of different threads - result.
Here's a variant with maximal load of all possible, maybe something better - I'll be glad.
I need functions:
forming a sequence of coefficients from an input sequence
merge of two sequences of coefficients into a single subresult (possible accuracy loss here)
merging of two sub-result into one slightly complete sub-result (there is a possible loss of accuracy)
I think it is possible to invent a clever partial loading mechanism, but it has to be invented.
For example, on the first read, find for each pass the last trade closed before the Start Date, go back and read X previous trades, remember the point in the file where that trade ends.
After that, find the first deal in the results, and then work only with fresh data: read the file from the desired point to the new actual date, and each time shift deals in the array (we will have a fixed size array - X elements).
This will solve the problem with multiple reads (we just don't need it) and memory (only if we can place X million transactions).
Yes, that's what the algorithm will do.
If we manage to allocate memory for X million trades(the size of the structure is known in advance), then we will be able to read the file once.
If not, then we need to add reading of the last X deals at each return to the Stored pointer. Then the file will be read multiple times but still economically.
The structure of the sequence will be fixed: Nos, Trades[X], Criterion, FilePointer position. There is nothing unnecessary.
All that remains is to code =)
What is the more desirable result:
dll or still using mql to calculate?
Yes, in this form the task is parallel - each timethe SeekDate changes,you can run a simultaneous search for the best Criterion on different parts of the sequence set. For example, we divide them into 20 parts and give the task to 20 Expert Advisors. And they should read the file, find the deal, and send back only the best sequence (№№, Criterion and file position).
Thank you all very much!
What is the more desirable result:
dll or still mql to calculate?
I haven't been with mql for half a year, I might be a little bit dumb, clarify if I'm wrong:
Открываем файл, начинаем читать, последовательно заполняя массив сделок первого прохода
are you planning to do a separate read from disk for each pass? 10^6 times read from disk?
Can't there be a hitch in reading individually instead of reading a whole chunk at once? Or is everything implemented at the highest level here with solid buffering to spare?
If we reach a trade, the opening time of which is >= SeekDate (it is not added to the array):
I don't see where the memory is being freed, it keeps piling up.
Why keep the whole array for one Criterion? You can simply compare them when calculating a new criterion and keep the appropriate one.
If you want to find 10 best criteria, it is better to make an array of 10 criteria, fill it with values, sort it, and then insert the next criteria using binary search.
Are you planning to perform a separate read from disk for each pass? Read 10^6 times from disk?
We will have to read the whole file anyway. We can't do it all at once (from beginning to end), so we'll have to read in pieces (but we'll still end up with the whole file).
Can't there be a hitch in reading one piece at a time instead of reading the whole piece at once? Or is everything implemented at the highest level here with solid buffering to spare?
A chunk is read. The size of the chunk is determined by the number of transactions before the Seeking Date, which were in a particular sequence.
I don't see where the memory is being freed, it keeps piling up.
The memory is allocated once for an array of sequence structures.
The sequence structure includes: No., Array of structures of all transactions of the sequence [X], Criterion value, File Pointer position.
The next step is just filling of the structure elements (including arrays of deals). The deals in the array are shifted, so there are always only X deals of each sequence in memory.
If you want to find 10 best criteria, it's better to create an array of 10 criteria, fill it with values, sort it, and then insert the next criteria using binary search.
Including for paralleling.
But eliminating one double from a structure that contains an array of deal structures doesn't make a difference within the framework of the task.
Sharing the results of my research.
Binary cache file of 7529 MB reads:
Conclusion: speedup on reading one big file using RAM disk is about 2.5 times.
If the file were sorted by time of deals not within sequences, but globally (by all sequences), then the following algorithm would be possible:
- Calculate the criterion for a trade, consider it a candidate
- We calculate criteria for deals that start inside this deal, if we get the best one, we change the candidate, if we don't get it, we consider the candidate selected and start a new cycle from the date of its closing.
We can also sort by close time - in this case we start from the end
It is clear that for calculation of a criterion, the file must contain sequence number for each trade.
Re-sorting of such a file is probably not a fun thing either, we can try to write it "properly" at once. That is not to generate whole sequences one by one, but to generate one transaction for each sequence and use some cache with intelligence when writing. For some generation algorithms, of course, this may be unacceptable.
If the file were sorted by time of deals not within sequences, but globally (by all sequences), then such an algorithm would be possible:
- Calculate the criterion for the deal, consider it a candidate
Suppose I record such a file (just converting the current one, even if it takes 15 hours of computer time).
But then - on the first point - there's a hitch. How do I calculate the Criterion on the last X trades of the sequence, having such a file?
Again, the Criterion cannot be calculated once, its parameters may change.