Problem with strings/StringConcatenate() in v600/v602 ?

gchrmt4 2014.02.06 07:41

Got the following code for reading a URL. However, there seems to be some sort of bug with string memory in both v600 and v602.

The first time I run this script after starting MT4, the string return value is corrupted. On all subsequent runs, the script works correctly.

A log of what's happening inside the function shows that the problem seems to be with StringConcatenate(). Each individual read (strThisRead) of data is okay, but the corruption starts occurring on the combined data (strData) on the second or later use of StringConcatenate(). Other methods of combining strings seem to make no difference. It's clearly memory corruption because the bad return values often include things like bits of names of network adapters.

I can't see any problem in the script which might make it the cause of the memory corruption.

#define READURL_BUFFER_SIZE   1000

#import "wininet.dll"
   int InternetOpenW(string, int, string, string, int);
   int InternetOpenUrlW(int, string, string, int, int, int);
   int InternetCloseHandle(int);
   int InternetReadFile(int, uchar & arr[], int, int & arr[]);
#import

                
// Reads the specified URL and returns the server's response. Return value is 
// a blank string if an error occurs                
string ReadUrl(string Url, bool PrintDebuggingMessages = false)
{
   string strData = "";
   bool bSuccess = false;
   
   // Get an internet handle
   int hInternet = InternetOpenW("mt4", 0 /* 0 = INTERNET_OPEN_TYPE_PRECONFIG */, NULL, NULL, 0);
   if (hInternet == 0) {
      if (PrintDebuggingMessages) Print("InternetOpenW() failed");

   } else {
      // Get a URL handle
      int hInternetUrl = InternetOpenUrlW(hInternet, Url, NULL, 0, 0, 0);
      if (hInternetUrl == 0) {
         if (PrintDebuggingMessages) Print("InternetOpenUrlW() failed");

      } else {
         if (PrintDebuggingMessages) Print("Okay: url handle: ", hInternetUrl);

         // We potentially (in fact, usually) get the response in multiple chunks
         // Keep calling InternetReadFile() until it fails, or until
         // it says that the read is complete
         bool bKeepReading = true;
         while (bKeepReading) {                    
            int szRead[1];
            uchar arrReceive[];
            ArrayResize(arrReceive, READURL_BUFFER_SIZE + 1);
            int success = InternetReadFile(hInternetUrl, arrReceive, READURL_BUFFER_SIZE, szRead);

            if (success == 0) {
               if (PrintDebuggingMessages) Print("InternetReadFile() failed");
               bKeepReading = false;

            } else {
               // InternetReadFile() has succeeded, but we may be at the end of the data 
               if (szRead[0] == 0) {
                  if (PrintDebuggingMessages) Print("Reached end of data");
                  bKeepReading = false;
                  bSuccess = true;
                  
               } else {
                  // Convert the data from Ansi to Unicode using the built-in MT4 function
                  string strThisRead = CharArrayToString(arrReceive, 0, szRead[0], CP_UTF8);
                  strData = StringConcatenate(strData, strThisRead);  // <-- PROBLEM HERE ON FIRST USE ONLY IN EACH MT4 SESSION
               }
            }
         }
         InternetCloseHandle(hInternetUrl);
      }
      InternetCloseHandle(hInternet);
   }

   return (strData);
}

// EXAMPLE:
void OnStart()
{
   string strData = ReadUrl("http://www.google.com/", true);
               
   Print(strData);
}

gchrmt4 2014.02.06 09:35 #1

gchrmt4:

Got the following code for reading a URL. However, there seems to be some sort of bug with string memory in both v600 and v602.

..The following does seem to work. It seems to "seed" the platform's memory handling:

void OnStart()
{
   ReadUrl("http://www.google.com/"); // Do a first call, and ignore the results

   string strData = ReadUrl("http://www.google.com/"); // Do a repeated call. Now seems to work.
   Print(strData);
}

Alain Verleyen 2014.02.06 14:17 #2

gchrmt4:

..The following does seem to work. It seems to "seed" the platform's memory handling:

Sorry but I don't understand what to problem is ?

gchrmt4 2014.02.06 14:26 #3

angevoyageur:
Sorry but I don't understand what to problem is ?

"The first time I run this script after starting MT4, the string return value is corrupted. On all subsequent runs, the script works correctly."

ydrol 2014.02.06 15:27 #4

I think you should build a testcase by making it read from a file instead of the internet? Maybe capture file first with wget etc?

esp as url is different for everybody according to location etc?

Then you can supply a file and code that demonstrates issue without calling features that will distract any support assistance :)

Window - Main menu Main menu Platform Installation - For

gchrmt4 2014.02.06 15:31 #5

ydrol:

I think you should build a testcase by making it read from a file instead of the internet? Maybe capture file first with wget etc?

Same effect on all URLs that I've tried (except for those where the server response is so short that there's only one call to InternetReadFile, and no string concatenation).

I'll have a look at reading a large file in chunks by passing a buffer to ReadFile().

Articles on the development Security System - For Trading Principles - Trade

gchrmt4 2014.02.06 15:53 #6

ydrol:

I think you should build a testcase by making it read from a file instead of the internet? Maybe capture file first with wget etc?

I'm seeing exactly the same kind of results with the following script which reads a file on disk rather than retrieving a URL. Again, the string is corrupt the first time that it is used after starting MT4, but works after that.

It makes no difference what the the file is: I have tried it with saved HTML output, a large text file just consisting of 01234567890123456789 etc

Either I am making the same class of mistake in both scripts, doing something which corrupts MT4's memory such as passing a buffer which is smaller than stated, or else there is a problem with string handling in MT4. I suspect the latter because otherwise the script should fail consistently, or at least more than once per MT4 session.

#define READ_BUFFER_SIZE   1000

#import "kernel32.dll"
   int CreateFileW(string, int, int, int, int, int, int);
   int GetFileSize(int, int);
   int ReadFile(int, uchar & buffer[], int, int & szRead[], int);
   int CloseHandle(int);
#import


string ReadTestFile(string strFileName)
{
   string strData = "";
   
   int FileHandle = CreateFileW(strFileName, -2147483648, 0, 0, 3, 0, 0);
   if (FileHandle == -1) {
      Print("Failed to open");
   } else {
      int FileSizeInBytes = GetFileSize(FileHandle, 0);
      
      int RemainingRead = FileSizeInBytes;
      while (RemainingRead > 0) {
         int ThisReadSize = MathMin(RemainingRead, READ_BUFFER_SIZE);
         
         uchar arrReadBuffer[];
         ArrayResize(arrReadBuffer, ThisReadSize);
         
         int szRead[1];
         if (!ReadFile(FileHandle, arrReadBuffer, ThisReadSize, szRead, 0)) {
            RemainingRead = 0;
            Print("File read error");
         } else {
            if (szRead[0] == 0){
               RemainingRead = 0;
               Print("File read error");
            } else {
               RemainingRead -= szRead[0];

               string strThisRead = CharArrayToString(arrReadBuffer, 0, szRead[0]);
               strData = StringConcatenate(strData, strThisRead);
            }
         }
      } 

      CloseHandle(FileHandle);
   }  

   return (strData);
}

// String output is corrupt the first time that the script is used during the MT4 session.
// Works successfully after that.
void OnStart()
{
   string strData = ReadTestFile(StringConcatenate(TerminalInfoString(TERMINAL_DATA_PATH), "\\MQL4\\Files\\url.txt"));
   Print(strData);
}

Example of developing a Working with Projects - Working with SQL data

gchrmt4 2014.02.06 16:16 #7

gchrmt4:

And, again, I'm seeing exactly the same issues with the following script which has no DLL calls at all. It's simply doing string concatenation. The function return value is corrupt when the script is first run during an MT4 session. It starts working after that.

This looks like a fundamental string-handling problem in MT4.

string TestConcatenation()
{
   string strTemplate;
   StringInit(strTemplate, 1000, 65);
   
   string strData = "";
   for (int i = 0; i < 10; i++) {
      strData = StringConcatenate(strData, strTemplate);
   }

   return strData;
}


void OnStart()
{
   string strData = TestConcatenation();
   Print(strData);
}

Example of developing a Migration - Virtual Hosting Average True Range -

Ubzen 2014.02.06 18:03 #8

gchrmt4:

And, again, I'm seeing exactly the same issues with the following script which has no DLL calls at all. It's simply doing string concatenation. The function return value is corrupt when the script is first run during an MT4 session. It starts working after that.

This looks like a fundamental string-handling problem in MT4.

If you believe StringConcatenate is the problem, why not isolate StringConcatenate?

string TestConcatenation()
{
   //string strTemplate;
   //StringInit(strTemplate, 1000, 65);
   
   string strData = "";
   for (int i = 0; i < 10; i++)
      strData = StringConcatenate(strData, "A");
   return strData;
}


void OnTick()
{
   string strData = TestConcatenation();
   Print("strData=",strData);
}

Why not show a print out of what you got and explain what you are expecting?

12:59:32 Expert TestingA EURUSD,H1: loaded successfully
12:59:33 Expert TestingA EURUSD,H1: loaded successfully
12:59:34 TestingA EURUSD,H1: initialized
12:59:34 TestingA EURUSD,H1: initialized
12:59:40 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:40 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:40 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:40 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:41 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:41 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:44 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:45 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:45 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:46 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:46 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:48 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:48 TestingA EURUSD,H1: strData=AAAAAAAAAA
12:59:49 TestingA EURUSD,H1: uninit reason 4
12:59:49 Expert TestingA EURUSD,H1: removed
12:59:53 TestingA EURUSD,H1: uninit reason 4
12:59:53 Expert TestingA EURUSD,H1: removed

I do-not use StringConcatenate nor StringInit. I usually just add+ them.

With StringConcatenate, I would expect the above.

Perhaps your problem is with StringInit?

Perhaps it needs additional time to initialize a 1000 string long As?

Perhaps this is more of a question of what the docs doesn't say?

I dunno.

Journal of Testing - Custom Financial Instruments - Margin Calculation: Retail Forex,

gchrmt4 2014.02.06 18:06 #9

ubzen:

12:59:45 TestingA EURUSD,H1: strData=AAAAAAAAAA

...May not apply to you, but there's independent confirmation that it's affecting other people: https://www.mql5.com/en/forum/149343

For absolute clarity, the steps to reproduce on all my test machines are as follows:

* Shut down all charts (because other EAs and indicators can potentially seed MT4's memory handling and mask the problem).

* Shut down MT4

* Restart MT4

* Open a chart

* Run the script

Lists of Objects Applied Task Manager - For Arrows - Analytical Objects

gchrmt4 2014.02.06 18:14 #10

ubzen:

If you believe StringConcatenate is the problem, why not isolate StringConcatenate?

See above: "Other methods of combining strings seem to make no difference"

Same results in v603.

1 2

New comment