Talend Open studio Component Installer ignores proxy in 7.2

Six Stars

Talend Open studio Component Installer ignores proxy in 7.2

Hi,

 

Like many of us, I live behind corporate proxy. In 7.0.1 the component installer works with proxy as expected. However, recently I tried to upgrade to 7.2.1 and utterly failed. I have proxy set at JVM level, at the application properties and in Java console. The component installer ignores all of it and still attempts direct connection. I verified this with Wireshark. 

 

Log entries for the failure.

 

!ENTRY org.talend.platform.logging 1 0 2019-07-30 10:54:17.371
!MESSAGE 2019-07-30 10:54:17,362 INFO org.talend.librariesmanager.librarydata.LibraryDataService - org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.postgresqlSmiley TongueostgresqlSmiley Tongueom:42.2.5 from/to talend (https://talend-update.talend.com/nexus/content/repositories/libraries/): Connection refused: connect


!ENTRY org.talend.platform.logging 1 0 2019-07-30 10:54:18.482
!MESSAGE 2019-07-30 10:54:18,473 INFO org.talend.librariesmanager.librarydata.LibraryDataService - org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact net.sourceforge.jtds:jtdsSmiley Tongueom:1.3.1-patch-20190523 from/to talend (https://talend-update.talend.com/nexus/content/repositories/libraries/): Connection refused: connect

 

Maybe there is a fourth way to set proxy that the LibraryDataService does respect?

 

I noted, that  the "Update componets" part seems to respect the proxy set in Java console, not the one in the env/preferences. However, nothing seems to apply to the single component installers. 

Moderator

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hello,

We are able to get these two jar files from talend update site via browser. Does this issue only block these two jar files?

jtds.pngpostgres.pngpostgresnexus.png

Best regards

Sabrina

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Yes, I can get them from browser too, if I manage to guess the URL as browser respects proxy setup. I didn't have time to do that. And my project needs in full a bit more than those 2... It's more than a little bit painful to guess full url of each and manually download. The proxy ignoring is problem not with those jars, but that down loader that does not seem to care about any proxy settings at all.

Moderator

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hello,

Not sure about how does your proxy setting look like. We will appreciate it a lot if you could give us some proxy setting screenshots on community. And please make sure there is no firewall or network issue.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hi,

My network blocks all non-proxy internet access. this is normal in corp intranet. Proxy is supposed to be used. All other ways that Talend Open Studio is accessing internet, including the connectivity check on startup and the "feature downloader" respect proxy setup and connect fine, however, this particular part, the artifact loader of the in-ui downloader attempts a direct connection. I have checked this with packet captures. All other Talend traffic goes to proxy, this particular function however does not. As my proxy configuration works for all other internet activity in  Talend, it must work.

I have set proxy in total 3 places, the JVM proxy options, the Preferences and in the Java Control Panel.

image.png

 

The first two duplicate each other as I can see in Talend startup logs that the UI prefrences are used to override the http.proxy* and https.proxy* parameters.

 

However, the component downloader/updater that pops up from the menu indicated

image.png

did not work and still attempted direct connections with that done. At that point, I configured the java control panel (picture below) with proxy instead of browser settings (there's apparently a known issue, that if the browser in turn is configured with PAC script that does not cascade to the java applications). This made the addional packages installer go to proxy, but fail to authenticate at proxy as it seems that proxy use method does not authenticate properly with windows integrated security for whatever reason. That I solved with my proxy managers and got to work.

 

image.png

With that out of the way only the in UI component downloader is left. With this there is nothing that can be done on proxy side as these artifact inventory info requests never go to proxy. Despite proxy being configured in any imaginable way and everything else using the proxy, it still does direct connection and fails...

 

In short, the artifact library access in org.talend.librariesmanager.librarydata.LibraryDataService class does HTTPS requests for POMs ignoring proxy. This I can not fix.

Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Unless there is a 4th proxy configuration hole somewhere...
Highlighted
Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

I dug through the relevant code and saw that this is passed down to Eclipse-s Aether library, that seems to be the one ignoring the proxy...
Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

setting global _JAVA_OPTIONS up with proxy information had no effect on this issue either.
Five Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hi,

 

I am facing the very exact same issue.

 

I would very much appreciate, if Talend could provide a detailed manuel on "how to set up a proxy connection in corporate environment". Including authentication issues, such as "NTLM", ...

 

best regards,

Chris

 

 

Tags (2)
Moderator

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hello @schnech1

There is no such a detailed manual on "how to set up a proxy connection in corporate environment" as the corporate environment is complex and uncertain.

Are you getting credential problem when configuring proxy setting?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hi,

It does not really matter how he configures it - the Eclipse base used in 7.2 has a bug in the proxy handling and no proxy configuration will take effect for certain library downloads... Unfortunately - you cant use 7.2 if you are behind a proxy. That was my conclusion. I'm still using 7.1 and hoping that the next version inherits from  eclipse a fixed proxy handling.

 

Best,

Thade 

Five Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2


@xdshi wrote:

Hello @schnech1

There is no such a detailed manual on "how to set up a proxy connection in corporate environment" as the corporate environment is complex and uncertain.

Are you getting credential problem when configuring proxy setting?

Best regards

Sabrina


Hi, unfortunately no. I only get log-entries such as this:
!ENTRY org.talend.platform.logging 1 0 2019-11-04 16:11:48.461 !MESSAGE 2019-11-04 16:11:48,461 INFO org.talend.librariesmanager.librarydata.LibraryDataService - org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.talend.libraries:jxlSmiley Tongueom:6.0.0 from/to talend (https://talend-update.talend.com/nexus/content/repositories/libraries/): connect timed out
@thade Proxy-configuration has been a pain in every talend version. I took me weeks to get it working in version 7.0.1. I will try version 7.1 and test, if it is working there.
Five Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Dear @thade and @xdshi ,

 

I just had a little session with our network specialist. A little summary.

 

We have a corporate environment with proxy (NTLM-Authentication), firewall and ssl-interception, scanning of internet traffic, ...

 

Version 7.0.1

To connect "Talend Data Integration" to the internet to be able to download additional components, we had to set an exception in the proxy. The proxy no longer asks for authentication, when trying to access the talend-update.talend.com URLs. With this rules, everything was fine.

 

Version 7.1

In addition to the above set of roules, we had to add "talend.com" to the list. I am not sure why, but now we are able to update Version 7.1 with additional components out of the talend repository.

 

Version 7.2

I can now confirm the statement of @thade : Talend Open Studio for Data Integration 7.2.1 does not use the proxy-configurations, but tries to access the internet directly. This is blocked by our network environment. (screenshot of our firewall)

 

firewall.png

 

 

 

 

Then there is something else: Talend sometimes asks me for the Community-Login. I added my credentials, but nothing seems to change. today, we have seen this error in the proxy-log. The proxy does not allow the connection to www.talendforge.org, because NTLM-Authentication does not work and because we did not add the url to the whitelist in the prosxy. 

connection: service.name=Explicit HTTP client.address=xxx.xxx.xxx.xxx proxy.port=xx client.interface=0:0.1 routing-domain=default
  location-id=0 access_type=unknown
time: 2019-11-08 08:24:10 UTC
GET http://www.talendforge.org/exchange/webservices/contributedExtension.php?data={"contributedExtension":{"passwordHash":"xxx","username":"schnech1"}}
  DNS lookup was unrestricted
User-Agent: Java/1.8.0_232
user: unauthenticated
authentication status='need_credentials' authorization status='not_attempted'
EXCEPTION(authentication_failed): Authentication failed either because credentials were not provided or they could not be validated

 

 

 

 

 

Moderator

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hello @schnech1 

Thanks for your summary. Does this issue only repro in V 7.2?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Talend Open studio Component Installer ignores proxy in 7.2

Hi @xdshi,

 

Hello @schnech1 

Thanks for your summary. Does this issue only repro in V 7.2?

Best regards
Sabrina

I assume you refere to my comment on the community login? No, this is related to Talend not being able to use proxy and NTLM-Authentication. This issue is present in all Talend versions 7.x. (As far as I know, the whole issue is based on the Eclipse version underneath Talend.)

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog