One Star

[ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Hello,
I am using TOS Big Data 5.5.1.r118616 - Hortonworks HDP sandbox 2.1 in virtualbox 
Windows 7 - 64bit
Iam doing a very simple
tHDFSconnection
    |
    V
tFixedFlowInput --> tFileOuptputDelimites
   |
   V
tHDFSPut
statistics] connecting to socket on port 3676
connected
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.(Shell.java:326)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.(Groups.java:77)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:256)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:233)
at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:310)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:304)
at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:534)
at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:348)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:244)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:144)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:579)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:524)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:154)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:154)
at tbd_551.t00_hdfs_connection_0_1.t00_hdfs_connection.tHDFSPut_1Process(t00_hdfs_connection.java:1391)
at tbd_551.t00_hdfs_connection_0_1.t00_hdfs_connection.tFixedFlowInput_1Process(t00_hdfs_connection.java:1289)
at tbd_551.t00_hdfs_connection_0_1.t00_hdfs_connection.tHDFSConnection_1Process(t00_hdfs_connection.java:607)
at tbd_551.t00_hdfs_connection_0_1.t00_hdfs_connection.runJobInTOS(t00_hdfs_connection.java:1846)
at tbd_551.t00_hdfs_connection_0_1.t00_hdfs_connection.main(t00_hdfs_connection.java:1616)
disconnected
Can you please help.
Amit

  • Big Data
7 REPLIES
Community Manager

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Hi Amit 
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

It seems you don't configure correctly the hadoop_home variable, take a look at a related topic:
http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binar...
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

hi, i´m studento of university Minho of Portugal.
I have a problem that talend and hadoop 2.3.
Exception in component tHDFSOutput_1
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
can help me
thank you
please bruno_martinho_1983@hotmail.com
One Star

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Not sure if the post resolves my issue - help appreciated
I am running Hadoop on AWS (EMR). I am using the Talend Studio to run a hadoop job. I assume I can run the job from my windows client?
I get the similar message 'Failed to locate the winutils binary in the hadoop binary path'
In the run dialog for the mapping I set the advanced setting -Dhadoop.home.dir=C:\AWS\Utils\winutils
I am only running the Talend client in windows (hadoop in emr) so not sure why I am getting this message or even if the software I have supports running this simple mapping
Thanks
FULL LOG...
Starting job SimpleJobMapR at 18:02 12/10/2016.
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
    at org.apache.hadoop.util.Shell.(Shell.java:364)
    at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:440)
    at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:486)
    at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)
    at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
    at myproject.simplejobmapr_0_1.SimpleJobMapR.runJobInTOS(SimpleJobMapR.java:4666)
    at myproject.simplejobmapr_0_1.SimpleJobMapR.main(SimpleJobMapR.java:4645)
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
connecting to socket on port 3690
connected
java.io.IOException: DataStreamer Exception:
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:574)
Caused by: java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1548)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1324)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1277)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)
disconnected
Job SimpleJobMapR ended at 18:04 12/10/2016.
Four Stars

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Hi brian,
I am getting similar error when I try to configure TOS 6.2 on my local windows machine and had hadoop on Cloudera VMWare.
I got the similar message 'Failed to locate the winutils binary in the hadoop binary path'. As per the information in http://www.srccodes.com/p/article/39/error-util-shell-failed-locate-winutils-binary-hadoop-binary-pa..., I need to add the winutils files into hadoop-common/bin folder.
But the hadoop is installed in my VMWare machine, so can u help me where do I need to add those files??
Below is the detailed error message from Talend Open Studio:
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:381)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:396)
at org.apache.hadoop.util.Shell.(Shell.java:389)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:130)
at org.apache.hadoop.security.Groups.(Groups.java:94)
at org.apache.hadoop.security.Groups.(Groups.java:74)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
at local_project.hdfs_load_0_1.hdfs_load.tHDFSDelete_1Process(hdfs_load.java:344)
at local_project.hdfs_load_0_1.hdfs_load.runJobInTOS(hdfs_load.java:879)
at local_project.hdfs_load_0_1.hdfs_load.main(hdfs_load.java:736)
disconnected
Five Stars

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Download 'winutils.exe' binary from: https:// github.com /steveloughran/winutils
Once downloaded, place in 'C:\winutil\bin'.  You may need to set this file and its path in your job's extra java arguments too.
https://help.talend.com/display/KB/The+missing+winutils.exe+program+in+the+Big+Data+Jobs?_ga=1.25783...
Four Stars

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Download 'winutils.exe' binary from: https:// github.com /steveloughran/winutils
Once downloaded, place in 'C:\winutil\bin'.  You may need to set this file and its path in your job's extra java arguments too.

Thanks Justin and it is working...!!
One Star

Re: [ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils

Hi Team,
I am also facing a similar issue related to "hadoop.util.NativeCodeLoader" and "LoginException:". I am tryng to establis a HDFS connection in Talend 6.3 Bigdata.


please suggest a solution.