One Star

handling redirect in tFileFetch

Hi, I'm trying to use tFileFetch in order to download https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
In a browser it first gives me back a HTTP/1.1 301 Moved Permanently and then redirects me to the download and sends the file right after.
The problem is that tFileFetch refuses to download that file. How can I accomplish that?
Starte Job tickdownloader am 00:44 30/09/2016.
connecting to socket on port 3985
connected
There isContinue_tFileFetch_1 an exception on: https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 301 Moved Permanently
at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

disconnected
Job tickdownloader endet am 00:44 30/09/2016.


Thank you in advance for your advice
2 REPLIES
One Star

Re: handling redirect in tFileFetch

Can you try fetching from the location where you are redirected?
One Star

Re: handling redirect in tFileFetch

Can you try fetching from the location where you are redirected?

I've tried it, but the server insists on the redirect:
There isContinue_tFileFetch_1 an exception on: http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 405 Not Allowed
at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

For now, I'm playing a bit with this (execute externally), but there must be a "Talend way" to do this ... plus: there is still the referrer problem if I use the result of this:
$ URL=https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
$ curl -w "%{url_effective}\n" -I -L -s -S $URL -o /dev/null
http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5