One Star

HDFSPut

I have a environment with a cloudera hadoop vmware server and a windows laptop with talend open studio. I want to copy files into HDFS from my windows environment and have tried to use the HDFSput component. The connection works fine, and im able to browser the HDFS from my workstation. But when I try to copy in files I get this error :
connecting to socket on port 4024
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception:
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:708)
Caused by: java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1622)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1420)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1373)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutp
Do I miss any library files on my workstation, or is it not possible to copy files to the HDFS from my workstaion.
Br
Petter
8 REPLIES
One Star

Re: HDFSPut

I chaged the port number in my connection from 8020 to 50010, and the file is created but it's empty, here are the output
connecting to socket on port 3993
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSPut_1
java.io.EOFException: End of File Exception between local host is: "NOHUSEBPET01/192.168.136.1"; destination host is: "192.168.181.128":50010; : java.io.EOFException; For more details see: 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy7.mkdirs(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539)
disconnected
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2758)
    at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2729)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
    at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
    at local_project.uu_0_1.uu.tHDFSPut_1Process(uu.java:442)
    at local_project.uu_0_1.uu.runJobInTOS(uu.java:819)
    at local_project.uu_0_1.uu.main(uu.java:664)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(Unknown Source)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Job uu ended at 22:16 28/11/2015.
One Star

Re: HDFSPut

Is this solved ?
Employee

Re: HDFSPut

Hello,
Can you confirm that the machine you are running the job on can access the machine "192.168.181.128" on the port 50010 ? It looks to be a firewall issue. Also, can you double check that the machine your are running the job on can ping "192.168.181.128" ?
One Star

Re: HDFSPut

I have the same issue using hortonworks instead cloudera, and yes the port 50010 is open, all ports have been opened in the firewall, and I can make a telnet to that port.
One Star

Re: HDFSPut

Was this solved? i have the same issue.
One Star

Re: HDFSPut

Check to see if the user account you are using has write permission to the folder you are writing.
One Star

Re: HDFSPut

 i have the same issue
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception: 
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1752)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1530)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1483)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:796)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1752)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1530)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1483)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
Fifteen Stars

Re: HDFSPut

Hi yepeng2007fei, your issue may be caused by this....
https://www.talendforge.org/forum/viewtopic.php?id=50662
Rilhia Solutions