handling redirect in tFileFetch

One Star

handling redirect in tFileFetch

Hi, I'm trying to use tFileFetch in order to download https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
In a browser it first gives me back a HTTP/1.1 301 Moved Permanently and then redirects me to the download and sends the file right after.
The problem is that tFileFetch refuses to download that file. How can I accomplish that?
Starte Job tickdownloader am 00:44 30/09/2016.
connecting to socket on port 3985
connected
There isContinue_tFileFetch_1 an exception on: https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 301 Moved Permanently
at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

disconnected
Job tickdownloader endet am 00:44 30/09/2016.


Thank you in advance for your advice
One Star

Re: handling redirect in tFileFetch

Can you try fetching from the location where you are redirected?
One Star

Re: handling redirect in tFileFetch

Can you try fetching from the location where you are redirected?

I've tried it, but the server insists on the redirect:
There isContinue_tFileFetch_1 an exception on: http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
java.lang.Exception: Method failed: HTTP/1.1 405 Not Allowed
at local_project.tickdownloader_0_1.tickdownloader.tFileFetch_1Process(tickdownloader.java:504)
at local_project.tickdownloader_0_1.tickdownloader.runJobInTOS(tickdownloader.java:907)
at local_project.tickdownloader_0_1.tickdownloader.main(tickdownloader.java:764)

For now, I'm playing a bit with this (execute externally), but there must be a "Talend way" to do this ... plus: there is still the referrer problem if I use the result of this:
$ URL=https://www.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5
$ curl -w "%{url_effective}\n" -I -L -s -S $URL -o /dev/null
http://datafeed.dukascopy.com/datafeed/USA30IDXUSD/2016/07/12/15h_ticks.bi5

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads