Error in HDFSOutput outside Open Studio in Linux/cron

Six Stars SJ
Six Stars

Error in HDFSOutput outside Open Studio in Linux/cron

Hi,

I am pulling some log files and putting it under a Hadoop folder. I can run this job in the big data open studio but when I try to run it in cron/linux, it throws following error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDeltaSmiley WinkV
at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)

161.PNG

I thought may be this was because of different Hadoop versions so I checked I have following HDFS version:

HDFS2.7.1.2.4

then I tried to change the versions of Hadoop under HDFSOutput component

161.PNG

But I couldn't find the same version. Now I am not sure how I can make this job work on linux/cron. Any suggestions?

Thanks!

SJ


Accepted Solutions
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I am using version 6.3.1. And I was able to solve this problem by not using any context variable under HDFS components. I have also used same HDFS connections

in tHDFSOutput and tHDFSList, which I have created under Hadoop Cluster - Metadata options. Thanks for your help though Sabrina!

 

SJ


All Replies
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I guess no one had this issue before 

Moderator

Re: Error in HDFSOutput outside Open Studio in Linux/cron

Hello,

The bigdata job works perfectly fine in studio but not in cmd(.sh file)? Are you using tHDFSconnection and tHBASEconnection in the same Job? Could you please indicate on which talend build version you got this issue?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I am using version 6.3.1. And I was able to solve this problem by not using any context variable under HDFS components. I have also used same HDFS connections

in tHDFSOutput and tHDFSList, which I have created under Hadoop Cluster - Metadata options. Thanks for your help though Sabrina!

 

SJ