hdfs connection error

Highlighted
Six Stars

hdfs connection error

I have installed cloudera VM and using CDH 5.13.

I am able to connect it using putty.

In THdfsInput I can view the HDFS file after choosing browse button. See screentshot view1

But after running I am getting an error

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... "using builtin-java classes where applicable
Exception in component tHDFSInput_1 (frstjob)
java.nio.channels.UnresolvedAddressException"


Accepted Solutions
Thirteen Stars

Re: hdfs connection error

@yogeshmalekar,can you un-tick Use Datanode Hostname option in basic settings of tHDFSInput and let me know.

Manohar B
Thirteen Stars

Re: hdfs connection error

@yogeshmalekar,revert back that change and can you check that the Namenode URI.

 

i believe you might have used hdfs://localhost:8020..can you changed to IP address and let me know that is working or not?

Manohar B

All Replies
Thirteen Stars

Re: hdfs connection error

@yogeshmalekar,can you un-tick Use Datanode Hostname option in basic settings of tHDFSInput and let me know.

Manohar B
Six Stars

Re: hdfs connection error

After removing use datanode hostname I am getting following error

 

[statistics] connecting to socket on port 3524
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Failed to connect to /127.0.0.1:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 1 IOException, will wait for 2376.0871920513 msec.
[WARN ]: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Failed to connect to /127.0.0.1:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 2 IOException, will wait for 8722.387499479391 msec.
[WARN ]: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Failed to connect to /127.0.0.1:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 3 IOException, will wait for 6415.130237157275 msec.
[WARN ]: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Failed to connect to /127.0.0.1:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused: no further information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:658)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Could not obtain block: BP-333635372-127.0.0.1-1508779710286:blk_1073742791_1983 file=/user/cloudera/yogesh/story.txt No live nodes contain current block Block locations: DatanodeInfoWithStorage[127.0.0.1:50010,DS-0b99f70d-aeda-4a68-9d34-f0da1e794231,DISK] Dead nodes: DatanodeInfoWithStorage[127.0.0.1:50010,DS-0b99f70d-aeda-4a68-9d34-f0da1e794231,DISK]. Throwing a BlockMissingException
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Could not obtain block: BP-333635372-127.0.0.1-1508779710286:blk_1073742791_1983 file=/user/cloudera/yogesh/story.txt No live nodes contain current block Block locations: DatanodeInfoWithStorage[127.0.0.1:50010,DS-0b99f70d-aeda-4a68-9d34-f0da1e794231,DISK] Dead nodes: DatanodeInfoWithStorage[127.0.0.1:50010,DS-0b99f70d-aeda-4a68-9d34-f0da1e794231,DISK]. Throwing a BlockMissingException
Exception in component tHDFSInput_1 (frstjob)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-333635372-127.0.0.1-1508779710286:blk_1073742791_1983 file=/user/cloudera/yogesh/story.txt
at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1040)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1023)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1002)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-333635372-127.0.0.1-1508779710286:blk_1073742791_1983 file=/user/cloudera/yogesh/story.txt
at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1040)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1023)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1002)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954)
at java.io.DataInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.talend.fileprocess.UnicodeReader.<init>(UnicodeReader.java:25)
at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:77)
at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:93)
at local_project.frstjob_0_1.frstjob.tHDFSInput_1Process(frstjob.java:582)
at local_project.frstjob_0_1.frstjob.runJobInTOS(frstjob.java:980)
at local_project.frstjob_0_1.frstjob.main(frstjob.java:831)
[statistics] disconnected

Thirteen Stars

Re: hdfs connection error

@yogeshmalekar,revert back that change and can you check that the Namenode URI.

 

i believe you might have used hdfs://localhost:8020..can you changed to IP address and let me know that is working or not?

Manohar B
Six Stars

Re: hdfs connection error

I am using the ip address. I can connect in putty using that ip address.

Six Stars

Re: hdfs connection error

Thanks.. I was doing two errors. 

1) I was not giving port which is 8020 by default

2) Use DataNode HostName was checked.

 

Thanks for your help.

Thirteen Stars

Re: hdfs connection error

@yogeshmalekar,great,your issue resolved. Kudos also accepted.

Manohar B

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.