Error in HDFSOutput outside Open Studio in Linux/cron

Highlighted
Six Stars SJ
Six Stars

Error in HDFSOutput outside Open Studio in Linux/cron

Hi,

I am pulling some log files and putting it under a Hadoop folder. I can run this job in the big data open studio but when I try to run it in cron/linux, it throws following error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDeltaSmiley WinkV
at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)

161.PNG

I thought may be this was because of different Hadoop versions so I checked I have following HDFS version:

HDFS2.7.1.2.4

then I tried to change the versions of Hadoop under HDFSOutput component

161.PNG

But I couldn't find the same version. Now I am not sure how I can make this job work on linux/cron. Any suggestions?

Thanks!

SJ


Accepted Solutions
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I am using version 6.3.1. And I was able to solve this problem by not using any context variable under HDFS components. I have also used same HDFS connections

in tHDFSOutput and tHDFSList, which I have created under Hadoop Cluster - Metadata options. Thanks for your help though Sabrina!

 

SJ


All Replies
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I guess no one had this issue before 

Moderator

Re: Error in HDFSOutput outside Open Studio in Linux/cron

Hello,

The bigdata job works perfectly fine in studio but not in cmd(.sh file)? Are you using tHDFSconnection and tHBASEconnection in the same Job? Could you please indicate on which talend build version you got this issue?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars SJ
Six Stars

Re: Error in HDFSOutput outside Open Studio in Linux/cron

I am using version 6.3.1. And I was able to solve this problem by not using any context variable under HDFS components. I have also used same HDFS connections

in tHDFSOutput and tHDFSList, which I have created under Hadoop Cluster - Metadata options. Thanks for your help though Sabrina!

 

SJ

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now