Saving results of optimization in CSV file is failing in some passes

[Martin Bittencourt]

Hi,

I have an EA that performs trading tests. It is my desire to save customize results data into a CSV file. I found out the function that is called right after a test is double OnTester() , so I created the following code to write the results into a file:

for (int aaa = 0; aaa < 100 && !IsStopped(); ++aaa)
   {
      if (!GlobalVariableSetOnCondition(MW_I_GV_RES_FILE_WRITTING,1.0,0.0))
      {
         Sleep(200);
         
         continue;
      }
      
      //
      const string testID = _Symbol 
            + " " + getTimingStrategiesAsString(TimingStrategy) 
            + " A" + IntegerToString(TS_IndAlgo);
               
      int resFileHandle = FileOpen(_Symbol + ".csv"
            ,FILE_READ|FILE_WRITE|FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_CSV|FILE_COMMON);   
      
      if (resFileHandle == INVALID_HANDLE)
         Print("DEBUG error opening test file: ",GetLastError());
      else
      {
         string resultsToWriteToFile = 
               getTimingStrategiesAsString(TimingStrategy) 
                     + " A" + IntegerToString(TS_IndAlgo)
               + "\t" + IntegerToString(TS_IndPeriod) 
               + "\t" + finalProfitS
               + "\t" + rentabilityS
               + "\t" + totalInvested2S
               ;

         //
         string currResultsToWriteToFile = FileReadString(resFileHandle);

         if (!FileSeek(resFileHandle,0,SEEK_END))
         {
            Print("DEBUG: Error putting file pointer at end: ",GetLastError());
         }
         else
         {
            if (currResultsToWriteToFile == "") //No head found; first writting
            {
               Print("DEBUG: Writing head 1: "
                     ,FileWrite(resFileHandle,
                           _Symbol
                           )
                     ," bytes written");
   
               Print("DEBUG: Writing head 2: "
                     ,FileWrite(resFileHandle,
                           "Test"
                           + "\tPeriod"
                           + "\tProfit"
                           + "\tRentability"
                           + "\tCapital Usage"
                           )
                     ," bytes written");
            }
            
            Print("DEBUG: Writing: ",resultsToWriteToFile
                  ,"; ",FileWrite(resFileHandle,resultsToWriteToFile)," bytes written");
         }
         
         FileClose(resFileHandle);
      }
      
      if (GlobalVariableSet(MW_I_GV_RES_FILE_WRITTING,0.0) == 0)
      {
         Print("DEBUG: Error putting GV to default: ",GetLastError());
      }
      
      break;
   }

Note: the part where the results are gattered and processed was omitted because they seemed useless. 

With this code, I'm able to open a file and write most of the results, but frequently not all of them. Here is the result I got in the last attempt when testing in Symbol PETR3 with data organized in Excel for clearer view:

So, as you can see, for whatever reason many of the passes (20, 30, 40) simply weren't registred. And there is no standarized behaviour: if I run it again with the exact same parameters, I may have these 3 written and some others not. Only very, very rearely I get all the results I'm supposed to have.

Since the backtest is done with concurrent threads, I supposed the problem was due to concurring access to the CSV file: while one thread was using it, another tried to use only to fail and save nothing, so I added the use of GlobalVariables working as mutexes to avoid that (since the documentation says it has atomic access to be used in situations with concurrent threads). This is the code I gave above and, as you can see, it didn't work. 

So what am I missing here? :| Just to be clear, all passes are run fine, only the writting of some of them while optimizing are failing (if I execute the missing results individually, I get them correctly):

[maxcalcroft96]

I would look into frames in the mql5 documentation and get the information from the passes that way in OnTester(). There is documentation and articles explaining how to do so that are very useful.

https://www.mql5.com/en/book/automation/tester/tester_framenext

Martin Bittencourt:

Hi,

I have an EA that performs trading tests. It is my desire to save customize results data into a CSV file. I found out the function that is called right after a test is double OnTester() , so I created the following code to write the results into a file:

Note: the part where the results are gattered and processed was omitted because they seemed useless. 

With this code, I'm able to open a file and write most of the results, but frequently not all of them. Here is the result I got in the last attempt when testing in Symbol PETR3 with data organized in Excel for clearer view:

So, as you can see, for whatever reason many of the passes (20, 30, 40) simply weren't registred. And there is no standarized behaviour: if I run it again with the exact same parameters, I may have these 3 written and some others not. Only very, very rearely I get all the results I'm supposed to have.

Since the backtest is done with concurrent threads, I supposed the problem was due to concurring access to the CSV file: while one thread was using it, another tried to use only to fail and save nothing, so I added the use of GlobalVariables working as mutexes to avoid that (since the documentation says it has atomic access to be used in situations with concurrent threads). This is the code I gave above and, as you can see, it didn't work. 

So what am I missing here? :| Just to be clear, all passes are run fine, only the writting of some of them while optimizing are failing (if I execute the missing results individually, I get them correctly):

[Martin Bittencourt]

maxcalcroft96 #:

I would look into frames in the mql5 documentation and get the information from the passes that way in OnTester(). There is documentation and articles explaining how to do so that are very useful.

https://www.mql5.com/en/book/automation/tester/tester_framenext

Hi,

thanks for the tip. I gave a look and it seems a little bit too much for what I need :T moreover, while trying to see what was different, I noticed the use of OnTesterInit and Deinit and, while trying to use them to open and close my CSV as in the example, neither were called resulting in errors when processig the CSV file. Morever, when running a single test, I could check, as the documentation states, that such functions are for optimization cases, but the one I want to create (as in the code above) is to work both when doing optimization as well as when doing single tests.

[Martin Bittencourt]

To add a new information to the study of my case: I decided to try to spot where exactly was the problem happening and created a series of entries in my code above where, whenever an error would occur, a second file would be created. Doing that I found out that basically the entire code works fine! In fact, I decided to open a file and write inside of it the number of bytes supposedly written in the CSV file and the results were 100% correct despite the error occurring.

So I though: since FileWrite is returning success in every case, but when I open the CSV file, some supposedly written lines are missing, that's problably due to the process of writting to the file on the CPU+SSD's part that has a delay on its own; so a FileFlush will probably do the trick. So I added a call to that function only to get no improvements:

            //Print("DEBUG: Writing: ",resultsToWriteToFile
                  //,"; ",FileWrite(glResFileHandle,resultsToWriteToFile)," bytes written");
                  
            int resFileHandleX = FileOpen("Writting " + IntegerToString(TS_IndPeriod) + ".csv"
                  ,FILE_READ|FILE_WRITE|FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_CSV|FILE_COMMON);   
            
            if (resFileHandleX != INVALID_HANDLE)
            {
               FileWrite(resFileHandleX,FileWrite(glResFileHandle,resultsToWriteToFile), " bytes written");
               FileFlush(glResFileHandle);
               FileClose(resFileHandleX);
            }

So I'm lost again to where could the problem be. Remembering: I'm using Global Variables as mutexes here so that the test file will be opened, written, flushed and close before some other concurring EA from another thread does the same. And yet it seems the processing of writting to the file is still messed up.

[Stanislav Korotky]

Martin Bittencourt #:

So I'm lost again to where could the problem be. Remembering: I'm using Global Variables as mutexes here so that the test file will be opened, written, flushed and close before some other concurring EA from another thread does the same. And yet it seems the processing of writting to the file is still messed up.

You can't use global variables as mutexes! Every tester agent has its own environment with its own set of global variables. These processes are separate from the terminal process with its own global variables.

[Alain Verleyen]

Martin Bittencourt #:

Hi,

thanks for the tip. I gave a look and it seems a little bit too much for what I need :T moreover, while trying to see what was different, I noticed the use of OnTesterInit and Deinit and, while trying to use them to open and close my CSV as in the example, neither were called resulting in errors when processig the CSV file. Morever, when running a single test, I could check, as the documentation states, that such functions are for optimization cases, but the one I want to create (as in the code above) is to work both when doing optimization as well as when doing single tests.

It's not "too much", it's exactly what you need, it's documented and there are examples on the forum (posted by me, among others).

You are wasting your time trying to solve something which has already a clear solution.

[Stanislav Korotky]

Alain Verleyen #:

It's not "too much", it's exactly what you need, it's documented and there are examples on the forum (posted by me, among others).

You are wasting your time trying to solve something which has already a clear solution.

Actually, if he wants to do this "synchronization" without frames, he can. One of possible solutions is to use files in common folder - one file for each agent, then merge - I described such approach in one of articles (look at the section Moving on to a Parallel World):

Data exchange between instances is required for calculating space without duplicate points. As you remember, the list of processed points in each object is stored in the 'index' binary tree. It could be sent to the terminal in a frame, similarly to the results, but the problem is that the hypothetical combined registry of these lists cannot be sent back to testing agents. Unfortunately, the tester architecture supports controlled data transfer only from agents to the terminal, but not back. Tasks from the terminal are distributed to agents in a closed format.

Therefore, I decided to use only local agents and to save each group indexes to files in a shared folder (FILE_COMMON). Each agent writes its own index and can read the indexes of all other passes at any time, as well as add them to its own index. This can be needed during pass initialization.

In MQL, changes in the written file can be read by other processes only when the file is closed. Flags FILE_SHARE_READ, FILE_SHARE_WRITE and the FileFlush functions do not help here.

[fxsaber]

It might be easier to read the cache after optimization.

[Alain Verleyen]

Stanislav Korotky #:

Actually, if he wants to do this "synchronization" without frames, he can. One of possible solutions is to use files in common folder - one file for each agent, then merge - I described such approach in one of articles (look at the section Moving on to a Parallel World):

I know but is that a useful solution ? Why would someone do that when there is an API to do it "live" ?