[Solved] tHiveRow for Hadoop HA

One Star

[Solved] tHiveRow for Hadoop HA

Hi all,
I am trying to load a file from HDFS into a Hive table using tHiveRow component.
Talend BigData Studio 5.6
Hadoop Hortonworks 2.1 with High Availability feature
Under advanced settings for tHiveRow I have:
dfs.client.use.datanode.hostname    true
dfs.datanode.use.datanode.hostname    true
dfs.nameservices    HDPCLUSTER
dfs.ha.namenodes.HDPCLUSTER    no,nt
dfs.namenode.rpc-address.HDPCLUSTER.no    hdp:8020
dfs.namenode.rpc-address.HDPCLUSTER.nt    hdp1:8020
dfs.client.failover.proxy.provider.HDPCLUSTER    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
When I run the job it's throwin the following exception:
FAILED: RuntimeException java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
java.lang.RuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:222)
        at org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:278)
        at org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:344)
        at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:246)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:426)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:976)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:969)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:99)
        at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:172)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:346)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:230)
        at data_integration.hiveexample_0_1.HiveExample.tHiveRow_1Process(HiveExample.java:549)
        at data_integration.hiveexample_0_1.HiveExample.tLibraryLoad_1Process(HiveExample.java:383)
        at data_integration.hiveexample_0_1.HiveExample.runJobInTOS(HiveExample.java:801)
        at data_integration.hiveexample_0_1.HiveExample.main(HiveExample.java:682)
Caused by: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:456)
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:148)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:579)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:524)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:210)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:445)
        ... 30 more
Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://HDPCLUSTER/tmp/hive-hive/hive_2015-01-16_12-53-36_655_5380462070013163379-1
        at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
        ... 35 more

Any ideas why?
Thank you
One Star

Re: [Solved] tHiveRow for Hadoop HA

Solved by adding a tHiveConnection component and use this existing connection in tHiveRow

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads