tFileFetch Ignores tSetProxy

Six Stars

tFileFetch Ignores tSetProxy

I have a job with a tFileFetch component to upload files in the job. Our traffic has to go through a proxy, but the proxy used is dependent on the context we are running.

 

For Example: production context = use proxy sample.com; development context = use <none>

 

Due to this need I cannot use the proxy settings on the tFileFetch component and instead executing a tSetProxy first then going to tFileFetch after the tSetProxy completes. The tFileFetch component does not pick up the proxy settings and instead tries to connect directly without any proxy settings. However, if I set tSetProxy to socks it does attempt to connect to that proxy, but our proxy server is not socks so the authentication message fails.

Ten Stars

Re: tFileFetch Ignores tSetProxy

I suggest using cUrl ... install it on your OS. Way easier to implement web integration...
I switched using talend web components (not for message queues) to cUrl, because of cookies, sessions, certifications, follow redirects, put post get , etc... way more flexible and route stdout to talend for proces information.
Good luck
Seventeen Stars

Re: tFileFetch Ignores tSetProxy

tSetProxy sets the JVM default settings for the TCP stack and change the proxy settings. It is simply impossible for any component using the network to ignore that.

Are you sure the tSetProxy is in action before your tFileFetch is working?

Six Stars

Re: tFileFetch Ignores tSetProxy

I agree, but here we are with it ignoring these settings. Below is how my job is designed. The scenario I'm testing is quite simple now and it simply does not care about the proxy being set.

 

 

Seventeen Stars

Re: tFileFetch Ignores tSetProxy

The proxy settings are also dedicated to a particular protocol. Please take care your used protocol are part of the new settings.

Six Stars

Re: tFileFetch Ignores tSetProxy

They are both set to http.

Six Stars

Re: tFileFetch Ignores tSetProxy

I have tried several versions and I feel I should add some more diagnostics.

 

If I have a tSetProxy->tRestClient->tFileFetch the tRestClient goes to the proxy appropriately, but the tFileFetch still does not go to the proxy and instead attempts to connect directly.

 

I have also tried using java to explicitly assign the proxy to system.properties, but I get the same result as if I am using the tSetProxy. This leads me to conclude that the problem is definitely in the tFileFetch component.

Ten Stars

Re: tFileFetch Ignores tSetProxy

cUrl... =|;-))
Six Stars

Re: tFileFetch Ignores tSetProxy

Security limitations of our deployment server limit our use of curl and cannot us it in this circumstance.

Ten Stars

Re: tFileFetch Ignores tSetProxy

Really Im sorry for you but this 'security' thing made me laugh. Must be frustrating for you.
If you are a little bit into programming and java: use Apache HttpClient : http://hc.apache.org/httpcomponents-client-4.5.x/httpclient/examples/org/apache/http/examples/client...

Look into the manual creating custom Routines in Talend. Pretty easy.
Six Stars

Re: tFileFetch Ignores tSetProxy

Thanks. I have written several routines in Talend to do things Talend cannot do. (ie. Pagination Parser, hierarchical sorting, more useful samba calls, etc.) so I am no stranger to routines.

 

However, I decided to use the tRest component to do the file upload and literally just got it working and it actually obeys the JVM proxy settings so it solves my problem. Going to give that a continued test throughout the day and see what results I can get.

Ten Stars

Re: tFileFetch Ignores tSetProxy

Kudos for you, this is one of the things which is frustrating for me (I think many others), by design it could never have worked but you loose precious time on figuring out how things work and are set back to zero.

There should be a curl like component... maybe one day I'll build "The One" web component.
Six Stars

Re: tFileFetch Ignores tSetProxy

Just to update. I could not make using the tRest component work, so I'm stuck using the built in proxy setting on the component. Using this is a procedural nightmare on our jobs because we develop in a different proxy space than our production environment so we have to make sure we change setting before and after we do development.

 

Our goal is to completely use the context variables to easily move between development and prod, but in this case we cannot because it is a checkbox on the component that needs to be ticked on or off.

 

So this never got solved and still persists as a problem. If someone out there figured out how to upload files via tRest I would be very interested.