Six Stars

[resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Dear all
We are using TBD 5.2.2.
Hadoop Cloudera CDH4.1.3 .
the defaulth version in TBD for CDH4 is CDH4.0.1,so we import jar of 4.1.3 into the component configuration file.
Maybe it works,but get errors:
xception in thread "main" java.lang.Error: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at projectname111.datatransfer_0_1.DataTransfer.tMSSqlInput_1Process(DataTransfer.java:652)
at projectname111.datatransfer_0_1.DataTransfer.runJobInTOS(DataTransfer.java:835)
at projectname111.datatransfer_0_1.DataTransfer.main(DataTransfer.java:699)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:378)
disconnected
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:146)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:144)
at projectname111.datatransfer_0_1.DataTransfer.tMSSqlInput_1Process(DataTransfer.java:452)
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 17 more
Can anybody give some regustion?
Thanks very much!

Joe
1 ACCEPTED SOLUTION

Accepted Solutions
Six Stars

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi Sabrina
you are right,TBD5.3.0 can fix this issue. What's more,new version is downward compatible. I can import the old jobs and they still work,without any modification of BigData components.
Thanks very much!
Best Regards,
Joe
8 REPLIES
Moderator

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi,
Perhaps this forum can point you a light Forum 29511
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi
I have tried tHDFSOutput.
Deleted ComponentsCache,and got the same error.
I have imported all the cdh4.1.3 jar that this component needs to D:\Talend\DataIntegration\TOS_BD-r99271-V5.2.2\plugins\org.talend.designer.components.bigdata_5.2.2.r99271\components\tHDFSOutput.
Moderator

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi,
Perhaps something wrong with that. How about choosing the "Custom definition" option in the latest version Talend Open Studio for BigData(5.3.0 r101800).
tHDFSOutput.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi,
Did you try to make the original component work against your CDH4.1.3 before modifying it?
In 5.2.2, even if the dependencies are CDH4.0.1, the components work with CDH4.1.2. I have not tested with CDH4.1.3, but since it's a minor version of CDH, I guess it should work as well.
Anyway, depending to your error above, commons-cli is missing in your classpath. But please try to use the original component, else, we can't help you.
Six Stars

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

but 5.2.2 does not have a "custom" option!
Moderator

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi,
The "custom " option is our new feature which is only available in latest version(5.3.0 r101800), so far.
See the related jira bug https://jira.talendforge.org/browse/TDI-25583.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi Sabrina
you are right,TBD5.3.0 can fix this issue. What's more,new version is downward compatible. I can import the old jobs and they still work,without any modification of BigData components.
Thanks very much!
Best Regards,
Joe
Moderator

Re: [resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

Hi Joe,
Thanks for your feedback. Feel free post your issue on forum.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.