Talentd Big Data 5.5.1 on windows against cloudera 5.1.2 on linux

One Star

Talentd Big Data 5.5.1 on windows against cloudera 5.1.2 on linux

I'm trying to read a file from HDFS (cloudera 5.1.2). If I'm using the the same simple job while running talend on a linux based machine (in this case ubuntu) it's working fine. However, the same job is failing while running talentd big data on windows with the following error message.
The root cause seems to be
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
Anybody any idea how to fix this?

Starting job test at 17:06 01/10/2014.
 
connecting to socket on port 3517
connected
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
      at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
      at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
      at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
      at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
      at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
      at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
      at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
      at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
      at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
      at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
      at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2554)
      at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2546)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2412)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
      at test.test_0_1.test.tHDFSGet_1Process(test.java:434)
      at test.test_0_1.test.runJobInTOS(test.java:900)
      at test.test_0_1.test.main(test.java:747)
disconnected
Job test ended at 17:06 01/10/2014.
Seventeen Stars

Re: Talentd Big Data 5.5.1 on windows against cloudera 5.1.2 on linux

Hadoop jobs will not run under Windows. It is a bad idea to try this - also with some additional windows helpers - because you test your jobs not under real live conditions. I strongly suggest using a Linux workstation or send the job to test to a remote jobserver running under Linux (depends you are working with a enterprise edition).
Employee

Re: Talentd Big Data 5.5.1 on windows against cloudera 5.1.2 on linux

Hi,
Since the Apache team has added support for Hadoop on Windows, windows client are looking for the winutils tool on the client machine. If not found, it's not used, but works anyway in almost all the cases. Even if this stack is tagged as an error, you can ignore it. Read https://issues.apache.org/jira/browse/HADOOP-11003
HTH,
Rémy.