WebRequest(): how to handle HTTP 303 response downloading shared files from Google Drive?

 

Hello all,

Wondering if someone could point me in the right direction vis-a-vis an issue I'm facing with WebRequest() when attempting to download shared files from Google Drive, e.g. sample file linked below:

https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download 

Attached is the script I'm using to test this -- pretty much a straightforward "GET" request via WebRequest() call to the URL above... The issue seems to be that drive.google.com responds with HTTP 303 (See Other) which redirects to a server in a different domain (in this case doc-04-c8-docs.googleusercontent.com). So, basically, my questions are:

  1. Is there a simple and stable way to work around this issue on Google Drive, which does not require the user to keep adding an arbitrary number of allowed URLs to the list under MT5 Options | Expert Advisors?
  2. If not possible with Google Drive specifically, what other file hosting options I could look into that do not require complex HTTP request/response handling?


Here's a snapshot from the debugger after this piece of code:

int timeout = 5000;
string web_url = "https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download";
string result_headers, headers = "";
char result_data[], post_data[];
int response = WebRequest("GET", web_url, headers, timeout, post_data, result_data, result_headers);

Here are the full contents of the `results_headersstring for reference just in case:

"Content-Type: application/binary
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 23 Nov 2022 19:12:22 GMT
Location: https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/29objud691gdskn9gh75bkb4h4s97lvk/1669230675000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=6e0efd01-78de-427a-9184-2b1b27cc4c88
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: script-src 'report-sample' 'nonce-rqVzwkg7pZW9GfsVIwDGFA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Server: ESF
Content-Length: 0
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
" (length: 1494)


Here's a snapshot from wget fetching the same URL -- ideally, I'd like if WebRequest() could handle things in the same manner 😇

$ wget 'https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download'
--2022-11-23 19:08:49--  https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download
Resolving drive.google.com (drive.google.com)... 142.250.200.110, 2a00:1450:4003:808::200e
Connecting to drive.google.com (drive.google.com)|142.250.200.110|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/l9ah82ffg2h3f91degchsc1c45bqc1lo/1669230525000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=e94e63c7-e275-4899-b9ff-e2282738ebfd [following]
Warning: wildcards not supported in HTTP.
--2022-11-23 19:08:49--  https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/l9ah82ffg2h3f91degchsc1c45bqc1lo/1669230525000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=e94e63c7-e275-4899-b9ff-e2282738ebfd
Resolving doc-04-c8-docs.googleusercontent.com (doc-04-c8-docs.googleusercontent.com)... 142.250.201.65, 2a00:1450:4003:811::2001
Connecting to doc-04-c8-docs.googleusercontent.com (doc-04-c8-docs.googleusercontent.com)|142.250.201.65|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 54 [text/plain]
Saving to: ‘uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download’

uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBv 100%[================================================================>]      54  --.-KB/s    in 0s

2022-11-23 19:08:49 (1.21 MB/s) - ‘uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download’ saved [54/54]
Files:
 
  1. Unfortunately, not! Only by using DLL calls to Windows API instead. or using some intermediate web-application.
  2. I guess you will have to experiment and see! I've never considered doing such things with WebRequest().
 

Have you allowed the URL in the Options (Ctrl+O):


 
Carl Schreiber #: Have you allowed the URL in the Options (Ctrl+O):
Yes he has and that is exactly what he is referring too. Google drive generates different server address depending on situation, thus requiring manual updating of that very list of accepted URLs.
 
Fernando Carreiro #:
  1. Unfortunately, not! Only by using DLL calls to Windows API instead. or using some intermediate web-application.
  2. I guess you will have to experiment and see! I've never considered doing such things with WebRequest().

Hi Fernando -- thanks for the quick reply, as usual, albeit ultimately disappointing :-)

 
Dmitri Diall: Wondering if someone could point me in the right direction vis-a-vis an issue I'm facing with WebRequest() when attempting to download shared files from Google Drive, e.g. sample file linked below:

Just remembered a possibility — have you tried just setting "https://googleusercontent.com" in the URL list?

Maybe MetaTrader will automatically consider the "*.googleusercontent.com" sub-domains as valid.

I've never tried it before but maybe it works.

 

I just tried adding "https://*.googleusercontent.com" with a wildcard and it accepted it, but don't know if it actually works.


 

What about tiny Url: https://tinyurl.com/app/

TinyURL.com - shorten that long URL into a tiny URL
  • tinyurl.com
TinyURL is the original URL shortener that shortens your unwieldly links into more manageable and useable URLs.
 
Carl Schreiber #: What about tiny Url: https://tinyurl.com/app/

That will not work. It is not a fixed URL. We are talking about Google Drive here that applies load balancing by directing to different servers.

EDIT: Besides, "tinyurl" just redirects the URL to the correct one. It will suffer the same issue.

 
Fernando Carreiro #:

I just tried adding "https://*.googleusercontent.com" with a wildcard and it accepted it, but don't know if it actually works.


Good call on the wildcard... Just tried it, but doesn't seem to work -- it appears that the full domain name needs to be added to the allowed URL list.

int response   = WebRequest("GET", web_url, headers, timeout, post_data, result_data, result_headers);
int error_code = GetLastError();

The code above results in response == -1 and error_code == 4014 which, in MQL5, means "function is not allowed for call" (ERR_FUNCTION_NOT_ALLOWED), i.e. URL not found in allowed list.

Runtime Errors - Codes of Errors and Warnings - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

Documentation on MQL5: Constants, Enumerations and Structures / Codes of Errors and Warnings / Runtime Errors
Documentation on MQL5: Constants, Enumerations and Structures / Codes of Errors and Warnings / Runtime Errors
  • www.mql5.com
Runtime Errors - Codes of Errors and Warnings - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Dmitri Diall #: Good call on the wildcard... Just tried it, but doesn't seem to work -- it appears that the full domain name needs to be added to the allowed URL list. The code above results in response == -1 and error_code == 4014 which, in MQL5, means "function is not allowed for call" (ERR_FUNCTION_NOT_ALLOWED), i.e. URL not found in allowed list. Runtime Errors - Codes of Errors and Warnings - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

Did you try my suggestion of just using the parent domain name?

Forum on trading, automated trading systems and testing trading strategies

WebRequest(): how to handle HTTP 303 response downloading shared files from Google Drive?

Fernando Carreiro, 2022.11.23 22:24

Just remembered a possibility — have you tried just setting "https://googleusercontent.com" in the URL list?

Maybe MetaTrader will automatically consider the "*.googleusercontent.com" sub-domains as valid.

I've never tried it before but maybe it works.