topic Re: handling redirect in tFileFetch in Talend Studio

handling redirect in tFileFetch

Anonymous — Thu, 29 Sep 2016 23:48:49 GMT

Hi, I'm trying to use tFileFetch in order to download https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
In a browser it first gives me back a HTTP/1.1 301 Moved Permanently and then redirects me to the download and sends the file right after.
The problem is that tFileFetch refuses to download that file. How can I accomplish that?

Starte Job tickdownloader am 00:44 30/09/2016.
 connecting to socket on port 3985
 connected
There isContinue_tFileFetch_1 an exception on: https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 301 Moved Permanently
	at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
	at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
	at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

 disconnected
Job tickdownloader endet am 00:44 30/09/2016.

Thank you in advance for your advice

Re: handling redirect in tFileFetch

Anonymous — Fri, 30 Sep 2016 04:41:29 GMT

Can you try fetching from the location where you are redirected?

Re: handling redirect in tFileFetch

Anonymous — Fri, 30 Sep 2016 10:38:58 GMT

Can you try fetching from the location where you are redirected?

I've tried it, but the server insists on the redirect:

There isContinue_tFileFetch_1 an exception on: http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 405 Not Allowed
	at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
	at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
	at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

For now, I'm playing a bit with this (execute externally), but there must be a "Talend way" to do this ... plus: there is still the referrer problem if I use the result of this:

$ URL=https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
$ curl -w "%{url_effective}\n" -I -L -s -S $URL -o /dev/null
http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5