Five Stars

TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hi,

 

I'm using TOS Big Data 6.4. We've upgraded from HDP 2.5 to HDP 2.6 and many componentes that worked fine under HDP 2.5, are not working anymore. When you select the distribution HDP 2.6, it seems that Talend is not able to recognize which libraries to load, so it displays an error saying 'library_name cannot be resolved to a type'. However, if you select HDP 2.5, it uses the correct libraries (which are the same for both distributions) and it works properly.

So it seems like a bug in TOS 6.4 when you choose HDP 2.6

 

Any solution for this issue?

 

Thanks

5 REPLIES
Moderator

Re: TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hello,

Talend studio doesn't prompt you to download the libraries needed? Can you see the required jar from module? Does this issue only repro on pig, sqoop components?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hi Sabrina,

 

there are some componentes (hive) that recognize which libraries to use but this is not the case for sqoop and pig (these ones are the only ones which I found that doesn't work correctly until now). I can see the required jars from Module because, with Pig for example, it's the same version (0.16.0) for both distributions (HDP 2.5 and HDP 2.6), if I choose HDP 2.5 it works but if I choose HDP 2.6 it doesn't. There's not the usual prompt which tells you which one is the required jar. Instead of that, a wizard is displayed (I attach a screenshot), but it doesn't tell you which one specifically. If I load the jar with a tLibraryLoad, the previous jars are read but a new wizard is displayed saying more jars are missing and if I load them again with a new tLibraryLoad, new jars missing appear, it seems it never ends!

 

Regards

 

 

Moderator

Re: TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hi,

Are you using talend open studio bigdata 6.4.0? We will make a testing on your issue to see if it repro on our side.

Thanks for your time.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hi,

yes, that's the version we're using.

 

Thanks a lot.

 

Regards

Moderator

Re: TOS 6.4 with HDP 2.6 not finding libraries for some componentes (Pig, Sqoop)

Hello,

Here is a related jira issue:https://jira.talendforge.org/browse/TBD-5215.

It seems this issue is fixed on V 6.4.1. Could you please take a look at it?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.