I have a job with a tFileFetch component to upload files in the job. Our traffic has to go through a proxy, but the proxy used is dependent on the context we are running.
For Example: production context = use proxy sample.com; development context = use <none>
Due to this need I cannot use the proxy settings on the tFileFetch component and instead executing a tSetProxy first then going to tFileFetch after the tSetProxy completes. The tFileFetch component does not pick up the proxy settings and instead tries to connect directly without any proxy settings. However, if I set tSetProxy to socks it does attempt to connect to that proxy, but our proxy server is not socks so the authentication message fails.
tSetProxy sets the JVM default settings for the TCP stack and change the proxy settings. It is simply impossible for any component using the network to ignore that.
Are you sure the tSetProxy is in action before your tFileFetch is working?
I agree, but here we are with it ignoring these settings. Below is how my job is designed. The scenario I'm testing is quite simple now and it simply does not care about the proxy being set.
The proxy settings are also dedicated to a particular protocol. Please take care your used protocol are part of the new settings.
I have tried several versions and I feel I should add some more diagnostics.
If I have a tSetProxy->tRestClient->tFileFetch the tRestClient goes to the proxy appropriately, but the tFileFetch still does not go to the proxy and instead attempts to connect directly.
I have also tried using java to explicitly assign the proxy to system.properties, but I get the same result as if I am using the tSetProxy. This leads me to conclude that the problem is definitely in the tFileFetch component.
Thanks. I have written several routines in Talend to do things Talend cannot do. (ie. Pagination Parser, hierarchical sorting, more useful samba calls, etc.) so I am no stranger to routines.
However, I decided to use the tRest component to do the file upload and literally just got it working and it actually obeys the JVM proxy settings so it solves my problem. Going to give that a continued test throughout the day and see what results I can get.
Just to update. I could not make using the tRest component work, so I'm stuck using the built in proxy setting on the component. Using this is a procedural nightmare on our jobs because we develop in a different proxy space than our production environment so we have to make sure we change setting before and after we do development.
Our goal is to completely use the context variables to easily move between development and prod, but in this case we cannot because it is a checkbox on the component that needs to be ticked on or off.
So this never got solved and still persists as a problem. If someone out there figured out how to upload files via tRest I would be very interested.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Create systems and workflow to manage clean data ingestion and data transformation.