One Star

HDFS Output Data Streamer Exception

Hello, 
I am trying to build a simple job, like in the image below:
The Output file is created on HDFS cluster but no data is written.

But I keep having the errors below:
Starting job Test at 09:31 20/12/2016.
connecting to socket on port 3756
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ID FirstName LastName
1|Warren|Harrison
1|Bill|Eisenhower
1|Andrew|Johnson
1|Bill|Johnson
1|Franklin|Harding
1|Lyndon|Grant
1|Richard|Reagan
1|Benjamin|Wilson
1|George|Quincy
1|Gerald|Harding
1|Franklin|Pierce
1|William|Harrison
1|Grover|Johnson
1|William|Harrison
1|Ulysses|Fillmore
1|Gerald|Fillmore
1|Dwight|Fillmore
1|Jimmy|Johnson
1|Chester|Harrison
1|Richard|Taft
1|Thomas|Carter
1|Lyndon|Hayes
1|Gerald|Harrison
1|Thomas|Roosevelt
1|Andrew|Harrison
1|Thomas|Reagan
1|Theodore|Grant
1|William|Van Buren
1|Dwight|Arthur
1|Dwight|Pierce
1|Chester|Monroe
1|Lyndon|Truman
1|Lyndon|Taft
1|Woodrow|Monroe
1|Woodrow|Johnson
1|Rutherford|Harrison
1|Chester|Grant
1|Rutherford|Washington
1|Woodrow|Pierce
1|Calvin|Reagan
1|James|Buchanan
1|Ulysses|Nixon
1|Gerald|Reagan
1|Lyndon|Wilson
1|George|Johnson
1|Gerald|Adams
1|Rutherford|Reagan
1|Martin|Harrison
1|John|Coolidge
1|Dwight|Washington
1|Theodore|Taft
1|Benjamin|Hoover
1|Millard|Garfield
1|Jimmy|Clinton
1|Chester|Harrison
1|James|Coolidge
1|Warren|Ford
1|George|Buchanan
1|Harry|Adams
1|John|Adams
1|Abraham|Wilson
1|Jimmy|Truman
1|Calvin|Carter
1|Chester|Fillmore
1|Andrew|Fillmore
1|Andrew|Wilson
1|Martin|Tyler
1|Lyndon|Reagan
1|Lyndon|Polk
1|Ulysses|Hayes
1|Franklin|Carter
1|Ronald|Eisenhower
1|Lyndon|Kennedy
1|William|Eisenhower
1|Herbert|McKinley
1|Millard|Jefferson
1|Gerald|Truman
1|Abraham|Hayes
1|Woodrow|Tyler
1|Ulysses|Van Buren
1|James|Harrison
1|Benjamin|Cleveland
1|Lyndon|Roosevelt
1|Calvin|Harding
1|George|Reagan
1|Grover|Lincoln
1|Theodore|Reagan
1|George|Ford
1|Benjamin|Coolidge
1|Martin|Polk
1|John|Jefferson
1|Theodore|Ford
1|Zachary|Wilson
1|Franklin|Monroe
1|Theodore|Truman
1|Herbert|Buchanan
1|Grover|Jackson
1|Abraham|Adams
1|Andrew|Roosevelt
1|Zachary|Hayes
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1782)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1559)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1512)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
Exception in component tHDFSOutput_2
java.io.IOException: DataStreamer Exception: 
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:796)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1782)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1559)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1512)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
disconnected
: org.apache.hadoop.hdfs.DFSClient - Failed to close inode 5757474
java.io.IOException: DataStreamer Exception: 
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:796)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1782)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1559)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1512)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
Job Test ended at 09:31 20/12/2016.
HDFS connection looks like this:
Note: If I use the namenode instead of IP , I receive more errors.
Thank you!
3 REPLIES
Moderator

Re: HDFS Output Data Streamer Exception

Hi,
Can you confirm that the machine you are running the job on can access your HDFS connection successfully? All ports have been opened in the firewall, and can you make a telnet to that port successfully?
Have you tried to use hostname of your namenode in your HDFSOutput component? 
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: HDFS Output Data Streamer Exception

Hi Sabrina, 
I have telnet succesfully port 8022,8020.
When I use hostname of my namenode, in both of my HDFS components, I get the errors above:
 Starting job Test at 14:36 21/12/2016.
connecting to socket on port 3839
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSOutput_2
java.lang.IllegalArgumentException: java.net.UnknownHostException: namenode1.hcluster.su.ro.conti.de
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:708)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:651)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:171)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
at helloworld.test_0_1.Test.tRowGenerator_3Process(Test.java:971)
at helloworld.test_0_1.Test.tHDFSConnection_1Process(Test.java:400)
at helloworld.test_0_1.Test.runJobInTOS(Test.java:1538)
at helloworld.test_0_1.Test.main(Test.java:1395)
Caused by: java.net.UnknownHostException: namenode1.hcluster.su.ro.conti.de
... 21 more
disconnected
Job Test ended at 14:36 21/12/2016.
Moderator

Re: HDFS Output Data Streamer Exception

Hi,
Have you already checked document about:TalendHelpCenterSmiley Frustratedetting up Hadoop connection manually?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.