Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

Four Stars

Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

Hi,
I am trying to connect to HDFS (2.7.3) using Talend big data platform (version 6.3) and loading data from csv to hadoop, but it's throwing below error. Attached is the complete log for the same.
Could you please advise on this. [Note: I also downloaded winutils.exe file (got version 2.6.0 on google) and put it under Talend folder and have also set HADOOP_HOME environment variable but it's not working out]
 
19:50:30.412 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
 
AND
 
19:50:30.412 [main] DEBUG org.apache.hadoop.util.Shell - Failed to find winutils.exe
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
 
Appreciate your help!!
 
Regards,
Pragya
Moderator

Re: Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

Hello,

Could you please have a look at this related topic:https://community.talend.com/t5/Sandbox/ERROR-org-apache-hadoop-util-Shell-Failed-to-locate-the-winu... to see if it works?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

Hi,

I downloaded the winutil.exe file from the given URL and set JVM Arguments in advance setting of Talend job like below.
-Dhadoop.home.dir=C:\\winutil\\

So now the above error is resolved, but it has started giving me some different error. below is the same. Do you have any idea about it.


Mar 07, 2018 3:20:09 PM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
INFO: fs.default.name is deprecated. Instead, use fs.defaultFS
Mar 07, 2018 3:20:09 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15:20:09.282 [main] DEBUG o.a.hadoop.util.PerformanceAdvisory - Falling back to shell based
Mar 07, 2018 3:20:09 PM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
INFO: fs.default.name is deprecated. Instead, use fs.defaultFS
15:20:09.333 [main] DEBUG o.a.h.security.UserGroupInformation - PrivilegedAction as:admin (authSmiley FrustratedIMPLE) fromSmiley Surprisedrg.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
15:20:16.152 [main] DEBUG o.a.hadoop.util.PerformanceAdvisory - Both short-circuit local reads and UNIX domain socket are disabled.
15:20:16.183 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
Mar 07, 2018 3:20:17 PM org.apache.hadoop.io.retry.RetryInvocationHandler handleException
WARNING: Exception while invoking ClientNamenodeProtocolTranslatorPB.mkdirs over null. Not retrying because try once and fail.
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[FATAL]: poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old - tHDFSPut_1 Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3061)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3031)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1162)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1158)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1158)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1150)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.tHDFSPut_1Process(DataLoadToHive_old.java:378)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.runJobInTOS(DataLoadToHive_old.java:774)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.main(DataLoadToHive_old.java:608)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:4060)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1118)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1012)

Exception in component tHDFSPut_1
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:603)
[statistics] disconnected
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3061)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3031)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1162)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1158)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1158)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1150)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.tHDFSPut_1Process(DataLoadToHive_old.java:378)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.runJobInTOS(DataLoadToHive_old.java:774)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.main(DataLoadToHive_old.java:608)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:4060)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1118)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1012)

Regards,
Pragya

Moderator

Re: Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

Hello,

From this online user guide:TalendHelpCenter: Supported Hadoop distribution versions for Talend Jobs for V 6.3, HDP 2.7.3 is not list supported Hadoop distribution versions table.

Are you using a custom distribution in your tHDFSPut component?

 

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.