How to speed up big array reading? - page 2

New comment
 
Lativ_:

Hello 

I'm trying to read a news file on back test from 2000 until present.

Is there a way to read the Array quicker? because what I'm trying for now is painfully slow

As file I/O is often the largest bottleneck, have you tried reading all lines into memory (an array, or set of arrays if the data set is very large) and then performing the complex processing while in-memory?

Could be faster if you pre-fetch and cache into RAM then do the complex manipulations, but you would have to test out whether reading the file first in one loop is better.

Also file I/O can be affected by fragmentation - if you can place the file on a separate unencrypted file system and defragment it, it could also help (if you want to more info on this, "clustering factor" in Oracle gives a good explanation). Such disk I/O optimization techniques used to be a big deal some years back, but in this age of SSD drives it may be not be as effective now

 
R4tna C #:

As file I/O is often the largest bottleneck, have you tried reading all lines into memory (an array, or set of arrays if the data set is very large) and then performing the complex processing while in-memory?

Could be faster if you pre-fetch and cache into RAM then do the complex manipulations, but you would have to test out whether reading the file first in one loop is better.

Also file I/O can be affected by fragmentation - if you can place the file on a separate unencrypted file system and defragment it, it could also help (if you want to more info on this, "clustering factor" in Oracle gives a good explanation). Such disk I/O optimization techniques used to be a big deal some years back, but in this age of SSD drives it may be not be as effective now

I just discovered MQL5 allows parallel processing - that was another way we used to speed up processing of large datasets.


https://www.mql5.com/en/articles/197

Parallel Calculations in MetaTrader 5
Parallel Calculations in MetaTrader 5
  • www.mql5.com
Time has been a great value throughout the history of mankind, and we strive not to waste it unnecessarily. This article will tell you how to accelerate the work of your Expert Advisor if your computer has a multi-core processor. Moreover, the implementation of the proposed method does not require the knowledge of any other languages besides MQL5.
 
William Roeder #:
Stop doing the same things multiple times
Simplify your code

Wow

 
R4tna C #:

I just discovered MQL5 allows parallel processing - that was another way we used to speed up processing of large datasets.


https://www.mql5.com/en/articles/197

Thank you for this

 

If you read this and also learning mql5. Having to deal with a large Array looping? I do not recommend this way of looping. I have found the most easiest and efficient way using ROM and its OOP. Please read this article

https://www.mql5.com/en/articles/719

Building an Automatic News Trader
Building an Automatic News Trader
  • www.mql5.com
This is the continuation of Another MQL5 OOP class article which showed you how to build a simple OO EA from scratch and gave you some tips on object-oriented programming. Today I am showing you the technical basics needed to develop an EA able to trade the news. My goal is to keep on giving you ideas about OOP and also cover a new topic in this series of articles, working with the file system.
 
Lativ_ #:

If you read this and also learning mql5. Having to deal with a large Array looping? I do not recommend this way of looping. I have found the most easiest and efficient way using ROM and its OOP. Please read this article

https://www.mql5.com/en/articles/719

Could you elaborate please? Did you mean RAM rather than ROM?

The article I posted was not this one 

 
R4tna C #:

Could you elaborate please? Did you mean RAM rather than ROM?

The article I posted was not this one 

Yes I meant RAM.

I didn't know about  MQL5 CArrayObj class 

12
New comment