Talend 5.3.0 and Hortonworks 1.2.4

One Star

Talend 5.3.0 and Hortonworks 1.2.4

Hi Team,
I am using Talend 5.3.0 and trying to connect to a remote Hortonworks (version 1.2.4) cluster.
Talend 5.3.0 supports Hortonworks1.0 (Hive Server1) and Hortonworks1.2.0(Biometa) (Hiveserver 1 and 2).
There is no option to connect to Hortonworks 1.2.4 in Talend 5.3.0 (for Hive_Connection) component and hence I tried with both the available options Hortonworks1.0 (Hive Server1) and Hortonworks1.2.0(Biometa) (Hiveserver 1 and 2).
But it did not work. Is this due to Version Mismatch or some other issue. Below are the Errors that I am facing.
Error - for Hortonworks1.0 (Hive Server1) and Hortonworks1.2.0(Biometa) (Hiveserver 1)
---------------------------------------------------------------------------------------------------------
13/06/12 03:05:46 ERROR parse.SemanticAnalyzer: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table empdet
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:897)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:831)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
at org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:127)
at demo.hive_0_1.Hive.tHiveRow_1Process(Hive.java:793)
.--+----+---.
| tLogRow_1 |
|=-+----+--=|
|ID|Name|Age|
|=-+----+--=|
'--+----+---'
disconnected
disconnected
at demo.hive_0_1.Hive.runJobInTOS(Hive.java:1167)
at demo.hive_0_1.Hive.main(Hive.java:1016)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:791)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:777)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:889)
... 13 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 22 more
FAILED: Error in semantic analysis: Unable to fetch table empdet
13/06/12 03:05:46 ERROR ql.Driver: FAILED: Error in semantic analysis: Unable to fetch table empdet
org.apache.hadoop.hive.ql.parse.SemanticException: Unable to fetch table empdet
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1129)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
at org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:127)
at demo.hive_0_1.Hive.tHiveRow_1Process(Hive.java:793)
at demo.hive_0_1.Hive.runJobInTOS(Hive.java:1167)
at demo.hive_0_1.Hive.main(Hive.java:1016)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table empdet
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:897)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:831)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
... 11 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:791)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:777)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:889)
... 13 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 22 more
13/06/12 03:05:46 INFO ql.Driver: </PERFLOG method=compile start=1371006306494 end=1371006346882 duration=40388>
13/06/12 03:05:46 INFO ql.Driver: <PERFLOG method=releaseLocks>
13/06/12 03:05:46 INFO ql.Driver: </PERFLOG method=releaseLocks start=1371006346882 end=1371006346882 duration=0>
Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Unable to fetch table empdet
Job Hive ended at 03:05 12/06/2013.

Error - for the option Hortonworks1.2.0(Biometa) (Hiveserver 2)
-------------------------------------------------------------------------
Starting job HiveTest at 11:57 12/06/2013.

connecting to socket on port 3527
connected
13/06/12 11:57:34 WARN conf.HiveConf: hive-site.xml not found on CLASSPATH
13/06/12 11:57:35 INFO service.AbstractService: ServiceSmiley SurprisedperationManager is inited.
13/06/12 11:57:35 INFO service.AbstractService: ServiceSmiley FrustratedessionManager is inited.
13/06/12 11:57:35 INFO service.AbstractService: Service:CLIService is inited.
13/06/12 11:57:35 INFO service.AbstractService: ServiceSmiley SurprisedperationManager is started.
13/06/12 11:57:35 INFO service.AbstractService: ServiceSmiley FrustratedessionManager is started.
13/06/12 11:57:35 INFO service.AbstractService: Service:CLIService is started.
13/06/12 11:57:35 INFO hive.metastore: Trying to connect to metastore with URI thrift://54.227.249.78:10000
13/06/12 11:57:35 WARN security.UserGroupInformation: No groups available for user talend
13/06/12 11:57:55 WARN hive.metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:2563)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:2549)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:292)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:162)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:102)
at org.apache.hive.service.cli.CLIService.start(CLIService.java:66)
at org.apache.hive.service.cli.thrift.EmbeddedThriftCLIService.<init>(EmbeddedThriftCLIService.java:35)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:86)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at version.hivetest_0_1.HiveTest.tHiveConnection_1Process(HiveTest.java:290)
at version.hivetest_0_1.HiveTest.runJobInTOS(HiveTest.java:519)
at version.hivetest_0_1.HiveTest.main(HiveTest.java:387)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 19 more
13/06/12 11:57:55 INFO hive.metastore: Waiting 1 seconds before next connection attempt.
13/06/12 11:57:56 INFO hive.metastore: Connected to metastore.
13/06/12 11:58:16 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out
13/06/12 11:58:16 ERROR hive.log: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:647)
disconnected
at org.apache.hive.service.cli.CLIService.start(CLIService.java:67)
at org.apache.hive.service.cli.thrift.EmbeddedThriftCLIService.<init>(EmbeddedThriftCLIService.java:35)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:86)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at version.hivetest_0_1.HiveTest.tHiveConnection_1Process(HiveTest.java:290)
at version.hivetest_0_1.HiveTest.runJobInTOS(HiveTest.java:519)
at version.hivetest_0_1.HiveTest.main(HiveTest.java:387)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 17 more
Exception in component tHiveConnection_1
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
at org.apache.hive.service.cli.CLIService.start(CLIService.java:69)
at org.apache.hive.service.cli.thrift.EmbeddedThriftCLIService.<init>(EmbeddedThriftCLIService.java:35)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:86)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at version.hivetest_0_1.HiveTest.tHiveConnection_1Process(HiveTest.java:290)
at version.hivetest_0_1.HiveTest.runJobInTOS(HiveTest.java:519)
at version.hivetest_0_1.HiveTest.main(HiveTest.java:387)
Caused by: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:827)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:649)
at org.apache.hive.service.cli.CLIService.start(CLIService.java:67)
... 8 more
Job HiveTest ended at 11:58 12/06/2013.

I tried the option of adding Hive-site.xml into C:\Talend\lib and C:\Talend\lib\java as per one of the Talend Forum discussions (26142) but did not work. (I am not giving the URL here due to restrictions)
My Hive Thrift API port is open and running fine.
So please assist to resolve this issue. Let me know if you need more details. Thanks!
Moderator

Re: Talend 5.3.0 and Hortonworks 1.2.4

Hi,
For talend open studio for Big Data, there is an option "Custom" in distribution.
See the component reference tHDFSConnection and the related reference Connecting to a custom Hadoop distribution
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend 5.3.0 and Hortonworks 1.2.4

Hi,
Thanks.
But will Talend be compatible with advanced versions like Hortonworks 1.2.4 while until 1.2.0 is supported in Talend 5.3.0.
I am getting error in the connection step itself - Eg: tHDFSConnection works fine but when I use tHDFSCOPY and click on the button to specify the file name, I get the error1 'HDFS Connection Failure - Check Connection Parameters'. Please assist
Employee

Re: Talend 5.3.0 and Hortonworks 1.2.4

Hello,
I don't see any reason. Talend Open Studio for BigData 5.3.0 should connect to HDP 1.2.4 successfully. Looking at the release note of HDP, the project version has not changed. Your issue looks to come from somewhere else.
Which port do you try to connect on? The error you have with HiveServer 1 doesn't come from the mistmatch version at all.
Rémy.
One Star

Re: Talend 5.3.0 and Hortonworks 1.2.4

Hi
The solution for the following exception is
" Exception in component tHiveConnection_1
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
at org.apache.hive.service.cli.thrift.EmbeddedThriftBinaryCLIService.<init>(EmbeddedThriftBinaryCLIService.java:35)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:113)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at "
The root cause for the above issue are:
1.Wrong port number
2.Wrong connection mode
Solution is:
you have to select standalone mode of the Hive2 connection rather the embedded mode with defined JDBC port number of the hive server. (see the image 1 for reference )
Use Hive server port number rather then Hive metasrore port number (See image 2 for reference, which show the hortonworks Hive and Metastore port number)
Regards,
Vengat Maran