WebRequest to download ForexFactory News

 

Hi

‌My code to download FF news don't work anymore since this WE ! Anyone using webrequest to download news has this problem ??

‌result = 0 bytes downloaded.  No modification in the code since last week when it was always running well since months.

‌‌both webpages https://cdn-nfs.forexfactory.net/ff_calendar_thisweek.xml and http://www.forexfactory.com/ffcal_week_this.xml always working if paste in Internet Explorer.

string cookie=NULL,headers; char post[],result[]; int res=0; //--- to enable access to the server, you should add URL //--- in the list of allowed URLs (Main Menu->Tools->Options, tab "Expert Advisors"): string lURL="https://cdn-nfs.forexfactory.net/ff_calendar_thisweek.xml"; //avant 3/3/17 -> "http://www.forexfactory.com/ffcal_week_this.xml"; //--- Reset the last error code ResetLastError(); //--- Loading a html page from Google Finance int timeout=5000; //--- Timeout below 1000 (1 sec.) is not enough for slow Internet connection if(gExt_ForceLectureFichierManuel==false) { res=WebRequest("GET",lURL,cookie,NULL,timeout,post,0,result,headers); }

 
Goutagny Philippe:

Hi

‌My code to download FF news don't work anymore since this WE ! Anyone using webrequest to download news has this problem ??

‌result = 0 bytes downloaded.  No modification in the code since last week when it was always running well since months.

‌‌both webpages https://cdn-nfs.forexfactory.net/ff_calendar_thisweek.xml and http://www.forexfactory.com/ffcal_week_this.xml always working if paste in Internet Explorer.

string cookie=NULL,headers; char post[],result[]; int res=0; //--- to enable access to the server, you should add URL //--- in the list of allowed URLs (Main Menu->Tools->Options, tab "Expert Advisors"): string lURL="https://cdn-nfs.forexfactory.net/ff_calendar_thisweek.xml"; //avant 3/3/17 -> "http://www.forexfactory.com/ffcal_week_this.xml"; //--- Reset the last error code ResetLastError(); //--- Loading a html page from Google Finance int timeout=5000; //--- Timeout below 1000 (1 sec.) is not enough for slow Internet connection if(gExt_ForceLectureFichierManuel==false) { res=WebRequest("GET",lURL,cookie,NULL,timeout,post,0,result,headers); }


I have the same problem, result = 9 bytes downloaded with this content ("forbidden")
 
Sergio Putzu:

I have the same problem, result = 9 bytes downloaded with this content ("forbidden")
The forexfactory.net URL works OK for me. It looks like they have blocked some user-agents on the .com URL. I guess they want to move the RSS traffic over to the .net URL. I got the .com URL to work by changing the user-agent.
void OnStart()
  {
   string cookie=NULL, headers;
   string reqheaders="User-Agent: Mozilla/4.0\r\n";
   char post[],result[];
   int res;
   string url="http://www.forexfactory.com/ffcal_week_this.xml";
   ResetLastError();
   int timeout=5000;
   res=WebRequest("GET",url,reqheaders,timeout,post,result,headers);
   if(res==-1)
     {
      Print("Error in WebRequest. Error code  =",GetLastError());
      //--- Perhaps the URL is not listed, display a message about the necessity to add the address
      MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
     }
   else
     {
      //--- Load successfully
      PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));
      //--- Save the data to a file
      int filehandle=FileOpen("test.xml",FILE_WRITE|FILE_BIN);
      //--- Checking errors
      if(filehandle!=INVALID_HANDLE)
        {
         //--- Save the contents of the result[] array to a file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- Close the file
         FileClose(filehandle);
        }
      else Print("Error in FileOpen. Error code=",GetLastError());
     }
  }
 
Tim Morris:
The forexfactory.net URL works OK for me. It looks like they have blocked some user-agents on the .com URL. I guess they want to move the RSS traffic over to the .net URL. I got the .com URL to work by changing the user-agent.
void OnStart()
  {
   string cookie=NULL, headers;
   string reqheaders="User-Agent: Mozilla/4.0\r\n";
   char post[],result[];
   int res;
   string url="http://www.forexfactory.com/ffcal_week_this.xml";
   ResetLastError();
   int timeout=5000;
   res=WebRequest("GET",url,reqheaders,timeout,post,result,headers);
   if(res==-1)
     {
      Print("Error in WebRequest. Error code  =",GetLastError());
      //--- Perhaps the URL is not listed, display a message about the necessity to add the address
      MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
     }
   else
     {
      //--- Load successfully
      PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));
      //--- Save the data to a file
      int filehandle=FileOpen("test.xml",FILE_WRITE|FILE_BIN);
      //--- Checking errors
      if(filehandle!=INVALID_HANDLE)
        {
         //--- Save the contents of the result[] array to a file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- Close the file
         FileClose(filehandle);
        }
      else Print("Error in FileOpen. Error code=",GetLastError());
     }
  }


Thanks a lot Tim !

 
Goutagny Philippe:


Thanks a lot Tim !

Tim Morris:
The forexfactory.net URL works OK for me. It looks like they have blocked some user-agents on the .com URL. I guess they want to move the RSS traffic over to the .net URL. I got the .com URL to work by changing the user-agent.
void OnStart()
  {
   string cookie=NULL, headers;
   string reqheaders="User-Agent: Mozilla/4.0\r\n";
   char post[],result[];
   int res;
   string url="http://www.forexfactory.com/ffcal_week_this.xml";
   ResetLastError();
   int timeout=5000;
   res=WebRequest("GET",url,reqheaders,timeout,post,result,headers);
   if(res==-1)
     {
      Print("Error in WebRequest. Error code  =",GetLastError());
      //--- Perhaps the URL is not listed, display a message about the necessity to add the address
      MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
     }
   else
     {
      //--- Load successfully
      PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));
      //--- Save the data to a file
      int filehandle=FileOpen("test.xml",FILE_WRITE|FILE_BIN);
      //--- Checking errors
      if(filehandle!=INVALID_HANDLE)
        {
         //--- Save the contents of the result[] array to a file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- Close the file
         FileClose(filehandle);
        }
      else Print("Error in FileOpen. Error code=",GetLastError());
     }
  }

string url="https://nfs.faireconomy.media/ff_calendar_thisweek.xml"; update  

 
On the same topic. Once the file is downloaded, how do you work with it? Here is my file download:
void      WebData()
 {
   string cookie=NULL, headers;
   string reqheaders="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19042\r\n";
   char post[],result[];
   double r[];
   
   int res;
   string url="https://data.bitcoinity.org/export_data.csv?c=e&data_type=volume&r=hour&t=b&timespan=7d";
   ResetLastError();
   int timeout=5000;
   res=WebRequest("GET",url,reqheaders,timeout,post,result,headers);
   if(res==-1)
     {
      Print("Error in WebRequest. Error code  =",GetLastError());
      //--- Perhaps the URL is not listed, display a message about the necessity to add the address
      MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
     }
   else
     {
      //--- Load successfully
      PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));
      //--- Save the data to a file
      int filehandle=FileOpen("test.csv",FILE_WRITE|FILE_BIN);
      //--- Checking errors
      if(filehandle!=INVALID_HANDLE)
        {
         //--- Save the contents of the result[] array to a file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- Close the file
         FileClose(filehandle);
        }
      else Print("Error in FileOpen. Error code=",GetLastError());
     }

}

This works, I can open the file in notepad. Now how do I extract the data into a structure or even just an array, but I have no idea how to do this. I've searched the forum and read many examples.

int file = FileOpen("test.csv",FILE_READ|FILE_TXT|FILE_COMMON);

   if(file != INVALID_HANDLE)
   {
      while(!FileIsEnding(file) && !_StopFlag)
       { FileReadString(file); }

      FileClose(file);
   }

Code above doesn't work. Doesn't work if It's FILE_CSV. I tried FILE_BIN and FileReadArray...didn't work. I'm obviously not understanding how to work with files once they're downloaded. Any help would be appreciated.