unable to connect hadoop using Talend Bigdata

Seven Stars

unable to connect hadoop using Talend Bigdata

Hi,

 

I have installed Hadoop in windows machine and tried to connect from Talend big data open studio, but I am unable to connect.

I am getting 'Compilation' error as follows.

 

Is it related to any jar files to be downloaded??

I am unable to download some optional packages, it is failing and tried multiple times.

 

connection_issue.JPG

 

 

connection_issue2.JPG


Accepted Solutions
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

I tried for the custom distribution also, but still it is asking some libraries to download and unable to download.

At last I have installed Cloudera in VMware and tried same distribution in Talend. Now I am not getting any compile time error. It seems the issue is with custom distribution.

View solution in original post


All Replies
Employee

Re: unable to connect hadoop using Talend Bigdata

Hi,

 

    It seems some of the JARs are missing in your case. Could you please specify the version you are using and the distribution you are trying to connect?

 

    When you are adding the component for the first time, it will list all the required jars at the top of the component tab for that component to execute correctly. Could you please cross check which one is missing there and try to install it again?

 

     Sometimes, if you are having any restrictions to access internet, you will not be able to download the Jars seamlessly. Could you please check above items and let us know your findings with screen shots?

 

Warm Regards,

 

Nikhil Thampi

 

     

Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi Nikhil,

 

I am using Talend Open Studio for Big Data 7.0.1 version and new to Hadoop, downloaded and configured in windows machine. Hadoop version is 2.7.7 .

Version:

2.7.7, rc1aad84bd27cd79c3d1a7dd53204a8c3ee1ed3ac

 

 

NB: I am unable to download some optional packages and tried multiple times all the time download is failing.

Is there any other way to download those packages. I tried using Help->Install Additional Packages.

Employee

Re: unable to connect hadoop using Talend Bigdata

@xdshi might be having more idea in this area. Adding her to the loop.

Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,

Are you getting some certificate issue when download third-party libraries for your talend open studio for bigdata?

What's JDK version are you using? Oracle JDK 1.8? Please ensure JDK bin is included in your PATH environment variable and is being reached.

More information will be highly appreciated.

Best regards

Sabrina

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

First of all thank you for your valuable comments and i have listed my comments below,

 

1. I am using Oracle JDK 1.8.0 and configured 'bin' path in system environment variables.

2. I have downloaded many packages (both required and optional) by Help-> Install Additional Packages tab.

3. But still some optional packages are there which is unable to download (refer the attached logs and below snapshots).

4. My concern is about the compilation error (mentioned in my topic) which I am getting in tHDFSConnection component and I am not able to see any required library which needs to be download from the same component.

 

Please share your views on this.

 

NB: Hope there is no proxy configuration needed, cause I have already downloaded libraries without changing the same.

 

optional_packages.JPG1.jpgseems this error blocking other package downloads

It seems this error (above) blocking all other library download....
2.JPGlist of packages

--------------------------------------------------------------------------

Regards,

Vibin CT.

 

 

Highlighted
Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,

Do you have a proxy / firewall? Have you tried to restart your Talend Studio for certain additional packages to take effect?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

I don't have any firewall or proxy and tried by restarting Talend studio multiple times.

I have already downloaded many third party libraries (both required and optional), so will the proxy/firewall affect rest of the libraries?

I have tried the same process in two laptops by using the same Talend version (latest), but no luck.

Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,

Is there any issue with server availability(https://talend-update.talend.com/nexus/) from your side? Are you able to access the URL outside talend?

https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...

You can copy and paste URL on your browser and check whether you are able to access it. 

We will appreciate it a lot if you could post Error Log on forum

To show the Error Log view on the tab system, go to Window > Show views, then expand the General node and select Error Log.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

I am able to access those URLs which you provided.

I have downloaded around 70% of the libraries by disabling firewall but failed again and unable to download rest libraries.

Please refer the attached log file for error log.

 

error_3.JPGerror_4.JPG

Forteen Stars

Re: unable to connect hadoop using Talend Bigdata

@Vibin_CT,can you check the required jars are installed in your talend setup? in Module section you will find the all jars.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,

Does the issue 'apache-curator' block you to use TOS-BD? It is not necessary to download all optional modules before using TOS-BD. The required jars will be pop-up again as a download/install dialog when you need to use some detailed function.

Here exists a jira issue about:

https://jira.talendforge.org/browse/TUP-20803

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @manodwhb,

 

Could you please specify where exactly I can see installed jars in Talend?

Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

Thanks for the information. I think 'Apache-curator' is blocking other jars to download.

But I am unable to see any jars needed for tHDFSConnection component for 'Custom-Unsupported' Distribution option.

Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,

Did you follow up this online document about:TalendHelpCenter:Connecting to a custom Hadoop distribution to connect hadoop distribution 2.7.7?

Have you tried to connect hadoop with supported distribution by talend(2.5, 2.6) to see if it works?

Best regards

Sabrina

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

I tried for the custom distribution also, but still it is asking some libraries to download and unable to download.

At last I have installed Cloudera in VMware and tried same distribution in Talend. Now I am not getting any compile time error. It seems the issue is with custom distribution.

View solution in original post

Moderator

Re: unable to connect hadoop using Talend Bigdata

Hello,


@Vibin_CT wrote:

Hi @xdshi,

 

I tried for the custom distribution also, but still it is asking some libraries to download and unable to download.

At last I have installed Cloudera in VMware and tried same distribution in Talend. Now I am not getting any compile time error. It seems the issue is with custom distribution.


Thanks for your feedback. Is the connection hadoop 2.7.7 OK with you?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: unable to connect hadoop using Talend Bigdata

Hi @xdshi,

 

Yes. Now I am able to connect Hadoop (Cloudera VMware) using Talend Big Data. Appreciate your valuable comments.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now