HDP connection from windows

Four Stars

HDP connection from windows

Hi everyone, I installed talend BD on windows 7, and I have a connection to the HDP 2.4, where usually I connect to it through putty ssh. Now, I would like to connect to this server through Talend BD, 

 I take the configuration information from core-site.xml and hdfs-site.xml.

but usually I receive this problem:

Exception in component tHDFSGet_1 (job1)
org.apache.hadoop.net.ConnectTimeoutException: Call From lg-PC/@ip to @ip:50740 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=isilon.irit.fr/141.115.102.125:50740]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
	at org.apache.hadoop.ipc.Client.call(Client.java:1431)
	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
[statistics] disconnected
	at com.sun.proxy.$Proxy7.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy8.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2094)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2077)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:801)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:863)
	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:859)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:859)
	at projet1.job1_0_1.job1.tHDFSGet_1Process(job1.java:665)
	at projet1.job1_0_1.job1.tHDFSConnection_1Process(job1.java:421)
	at projet1.job1_0_1.job1.runJobInTOS(job1.java:1100)
	at projet1.job1_0_1.job1.main(job1.java:949)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=isilon.irit.fr/141.115.102.125:50740]
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:612)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:710)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
	at org.apache.hadoop.ipc.Client.call(Client.java:1397)
	... 23 more
Job job1 terminé à 15:32 17/01/2018. 

 

Eight Stars

Re: HDP connection from windows

Hello,

 

Connection timeout is mostly firewall issue. Please check firewall configuration between the lg-PC/172.16.3.125 and isilon.irit.fr:50740 (HDP location and computer with Talend).

 

Regards

Lojdr

Four Stars

Re: HDP connection from windows

Hello, thanks for your response
I verified the firewall configuration, but it's enabled with talend.
And I added the IP address details in the hosts file. it isn't sufficient ?

Regards
Hana
Eight Stars

Re: HDP connection from windows

Hello,

 

Try to connect to specific port at HDP server from W7 via telnet. Are you able to establish the connection?

 

Regards

Lojdr

Four Stars

Re: HDP connection from windows

Hello,
You have a reason I cannot connect through this port only through 22 which open the ssh. But I have take the port from the core-site.xml of hadoop files .

Regards
Hana
Eight Stars

Re: HDP connection from windows

I see that your job is trying to connect to 

141.115.102.125:50740

Are you able to connect to this IP and port from your computer? If only port 22 is allowed you will have to tunnel it somehow.

Four Stars

Re: HDP connection from windows

My computer is not able to connect to this IP and Port.
Eight Stars

Re: HDP connection from windows

Hello,
You need to arrange your network to be able to access the port where the service/daemon is running.

Regards
Lojdr
Four Stars

Re: HDP connection from windows

Hello everyone.

 

I have been trying for about 5 days now to setup a Hadoop connection from Talend installed on my laptop running windows.

 

The cluster (multi node) is on a Linux box. 

 

I would appreciate any help.

 

Thanks in advance

Moderator

Re: HDP connection from windows

Hello,

Are you getting error message when you setup a Hadoop connection from Talend?

Could you please refer to this document about:TalendHelpCenter: Setting up a Hadoop cluster connection?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: HDP connection from windows

First, when I try to create the Hadoop connection and run "Check Services" it fails with this message.

 

org.talend.designer.hdfsbrowse.exceptions.HadoopServerException: org.talend.designer.hdfsbrowse.exceptions.HadoopServerException: java.util.concurrent.TimeoutException
at org.talend.designer.hdfsbrowse.hadoop.service.check.AbstractCheckedServiceProvider.checkService(AbstractCheckedServiceProvider.java:57)
at

 

Second, when I try to run tHDFSExists I get this message

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.

 

Can you tell me please if its possible to connect from Windows to a multi-node cluster on Linux? If yes, can you please point me to any example. My cluster does have an edge node which interacts with the internal nodes.

 

Thank you.

 

Four Stars

Re: HDP connection from windows

can anybody tell me if windows is supported in HDP 2.5? Version 2.4.2 is supported and for version 2.5.0 there is no download and information available (yet) at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0-Win/index.html