One Star

[resolved] TOS writing data to Hadoop HDFS error

Dear Talend support,
We had error when writing to Hadoop HDFS. Using TOS, we:
1. could access the HDFS
2. successfully created a file
3. from TOS, it shows that we sent all data to HDFS. But the file TOS created was empty and we got error message as below.
 Starting job hdfsout at 20:30 20/05/2016.
 
connecting to socket on port 3484
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
        at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1622)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1420)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1373)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
Exception in component tHDFSOutput_1
java.io.IOException: DataStreamer Exception:
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:708)
Caused by: java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
        at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1622)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1420)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1373)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
disconnected
: org.apache.hadoop.hdfs.DFSClient - Failed to close inode 17723
java.io.IOException: DataStreamer Exception:
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:708)
Caused by: java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
        at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1622)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1420)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1373)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
Job hdfsout ended at 20:30 20/05/2016.
 
1 ACCEPTED SOLUTION

Accepted Solutions
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

Thanks Shong and Rohan.
It is connectivity issue and after we change to a different Id, it works. Thank you for your time.
Lan
8 REPLIES
Community Manager

Re: [resolved] TOS writing data to Hadoop HDFS error

Hi 
Caused by: java.nio.channels.UnresolvedAddressException
        at sun.nio.ch.Net.checkAddress(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)

It looks like a connection issue,  are you able to read data from HDFS with tHDFSInput?
Regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

Shong,
If we could not connect to Hadoop, how can we create empty file? We can change the file name to a different name and create a different file. Yet, content is empty.
Lan
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

Shong, 
Followed your suggestion, we did the read test. Got error too. Any suggestions?
Starting job HDFSInput at 13:59 25/05/2016.
 
connecting to socket on port 3744
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - Failed to connect to sandbox.hortonworks.com/10.0.0.4:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 1 IOException, will wait for 1439.7276636017177 msec.
: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - Failed to connect to sandbox.hortonworks.com/10.0.0.4:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 2 IOException, will wait for 4769.762006394212 msec.
: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - Failed to connect to sandbox.hortonworks.com/10.0.0.4:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 3 IOException, will wait for 11059.781468103138 msec.
: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - Failed to connect to sandbox.hortonworks.com/10.0.0.4:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
      at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:618)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
: org.apache.hadoop.hdfs.DFSClient - Could not obtain block: BP-595454498-172.16.137.143-1456768655900:blk_1073742615_1799 file=/tmp/maria_dev/data/geolocation.csv No live nodes contain current block Block locations: DatanodeInfoWithStorage Dead nodes:  DatanodeInfoWithStorage. Throwing a BlockMissingException
: org.apache.hadoop.hdfs.DFSClient - Could not obtain block: BP-595454498-172.16.137.143-1456768655900:blk_1073742615_1799 file=/tmp/maria_dev/data/geolocation.csv No live nodes contain current block Block locations: DatanodeInfoWithStorage Dead nodes:  DatanodeInfoWithStorage. Throwing a BlockMissingException
: org.apache.hadoop.hdfs.DFSClient - DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-595454498-172.16.137.143-1456768655900:blk_1073742615_1799 file=/tmp/maria_dev/data/geolocation.csv
      at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
Exception in component tHDFSInput_1
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-595454498-172.16.137.143-1456768655900:blk_1073742615_1799 file=/tmp/maria_dev/data/geolocation.csv
      at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
      at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
      at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
      at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
      at local_project.hdfsinput_0_1.HDFSInput.tHDFSInput_1Process(HDFSInput.java:541)
disconnected
      at local_project.hdfsinput_0_1.HDFSInput.runJobInTOS(HDFSInput.java:903)
      at local_project.hdfsinput_0_1.HDFSInput.main(HDFSInput.java:760)
Job HDFSInput ended at 13:59 25/05/2016.
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

My guess is that the user has permission issue on the cluster. I have had similar experience where I was able to create table in Hive which creates the data file but could not do the insert into the table that I created!
Please check if the user you are using in talend is able to write data into hdfs.
Thanks
Rohan
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

Thanks Shong and Rohan.
It is connectivity issue and after we change to a different Id, it works. Thank you for your time.
Lan
Fifteen Stars

Re: [resolved] TOS writing data to Hadoop HDFS error

If you have been getting this issue, it could be related to this....
https://www.talendforge.org/forum/viewtopic.php?id=50662
Rilhia Solutions
One Star

Re: [resolved] TOS writing data to Hadoop HDFS error

Thanks Shong and Rohan.
It is connectivity issue and after we change to a different Id, it works. Thank you for your time.
Lan

Hi Lan
Can you please detail out "changing to different Id". I am also getting same error.
Regards
Arpit
Six Stars

Re: [resolved] TOS writing data to Hadoop HDFS error

I was seeing a similar error trying to connect to my AWS Hadoop setup.  
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception: 
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:563)
Caused by: java.nio.channels.UnresolvedAddressException
I had set up a pre-defined hadoop connection that tested out fine during setup, but when writing to Hadoop from my sample workflow the file got created, but the data did not get written and I got the above error.  I confirmed that my namenode had host file entries for each machine in the setup and that my Talend Hadoop connection was not using IP address (as suggested by rhall), but still no luck.  Then it dawned on me that the AWS namenode instance has to be re-started after updating the host file (noob error).  Once I restarted the AWS namenode and restarted hadoop then the issue was resolved. The posts in this forum are very helpful, so thanks all for the info in these posts.