WebRequest is being cached and not allowing new webpage data to be viewed

New comment
 

Hello,

I have a news advisor that is loading a webpage and downloading the data. There are two options to choose from, FXstreet, and myFXbook. MyFxbook seems to be caching our request and only presenting us the same page, even though their website has changed data. 

I've tried setting the max-age or cache-control, but their cloudflare does not seem to want to accept these. FXstreet doesn't seem to be doing this for their website.

 I understand that technically Cloudflare does not have to honour such requests, but maybe there is a way I can present my request different to appear new without changing my IP.

Any ideas?

FXstreet

   string cookie=NULL,referer=NULL,headers;
   char post[],result[];
   string tmpStr="";
   StringReplace(st_date,".","");
   StringReplace(end_date,".","");
   string url="http://calendar.fxstreet.com/EventDateWidget/GetMini?culture="+LANG+"&view=range&start="+st_date+"&end="+end_date+"&timezone=UTC"+"&columns=date%2Ctime%2Ccountry%2Ccountrycurrency%2Cevent%2Cconsensus%2Cprevious%2Cvolatility%2Cactual&showcountryname=false&showcurrencyname=true&isfree=true&_=1455009216444";
   ResetLastError();
   WebRequest("GET",url,cookie,referer,10000,post,sizeof(post),result,headers);


FXbooks

   string cookie=NULL,referer=NULL;//,headers;
   char post[],result[];
   string tmpStr="";
   string url="";
   //string headers="cache-control: no-cache, no-store, must-revalidate;\r\n age:2600000;\r\n";
   string headers="cache-control: public, no-cache, s-maxage=0, max-age=0;\r\n pragma: no-cache;\r\n expires:0;\r\n";
   ResetLastError();
   WebRequest("GET",url,cookie,referer,10000,post,sizeof(post),result,headers);
 
blackbic:

Hello,

I have a news advisor that is loading a webpage and downloading the data. There are two options to choose from, FXstreet, and myFXbook. MyFxbook seems to be caching our request and only presenting us the same page, even though their website has changed data. 

I've tried setting the max-age or cache-control, but their cloudflare does not seem to want to accept these. FXstreet doesn't seem to be doing this for their website.

 I understand that technically Cloudflare does not have to honour such requests, but maybe there is a way I can present my request different to appear new without changing my IP.

Any ideas?

FXstreet


FXbooks


To me it looks like your header is malformed and incomplete.

If it works with your browser, do following: Open dev console of your browser and copy the request header from there.

It might be necessary to have some additional code to manage cookies and alike, but you will need to test that.

It took me quite some effort to replicate exact behaviour of my browser inside MQL, but it is for sure possible to make it work.
 
try to keep on changing the request URL by adding dummy URI parameter, e.g. uri?&t=87643677 (could be incrrasing timestamp or random number)

Good luck.
 

Try adding the referrer as their own homepage . 

Now if they get foxy and they notice you were not there , add another precceeding web request and go there.

Also add a user agent , prefer an iphone user agent because they make more money from ads targeting iphones (if they use google ads).

Also don't make 3 million requests a day , in case you are.
New comment