One Star

Connect Talend to Hive in Hortonworks Sandbox

HI all,
I am new in the forum, sorry if this post already exists, but looking around I could not find any solution it could help.
I am trying to connect talend to Hive. I am running Talend 6.1.0 on  a virtual machine Ubuntu 14.04 and the sandbox is the Hortonworks Sandbox 2.3.0. Both machine are guested on a Windows machine and both are on bridge.
I was able to transfer file on the HDFS using talend.
I am using the tHiveConnection and I input the following parameteres
Host "192.168.1.3" (The IP of my sandbox) Port "10000"
Database "default"
Then in Hadoop properties I have:
Set Resource Manager: "sandbox.hortonworks.com:8032"
Set Namenode URI : "hdfs://sandbox.hortonworks.com:8020"
Set resourcemanager scheduler address: "sandbox.hortonworks.com:8030"
Set jobhistory address: "sandbox.hortonworks.com:10020"
Note that sandbox.hortonworks.com is the hostname for HW IP and yes is on the hosts file.
When I run however the following error arise:
---------------------------------------------------------------------------------------------------------------------------------
Starting job hive_connection at 17:01 30/11/2015.

connecting to socket on port 4031
connected
Exception in component tHiveConnection_1
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime
    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
    at hdfs.hive_connection_0_1.hive_connection.tHiveConnection_1Process(hive_connection.java:339)
    at hdfs.hive_connection_0_1.hive_connection.runJobInTOS(hive_connection.java:636)
    at hdfs.hive_connection_0_1.hive_connection.main(hive_connection.java:493)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
    at org.apache.hive.service.cli.operation.HiveCommandOperation.runInternal(HiveCommandOperation.java:108)
disconnected
    at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
    at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
    at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Job hive_connection ended at 17:01 30/11/2015.
--------------------------------------------------------------------------------------------------------------------------------------------
Can anyone help?

  • Talend Studio
17 REPLIES
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

I am also facing the same issue but with v5.6.2 of the Talend Big Data Studio.
Moderator

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi,
We have a jira issue about this exception "Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime"
Could you please take a look at it?
https://jira.talendforge.org/browse/TBD-622
Best regards
Sabrina
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Thank you for your reply, but it does not seem that the problem is solved. I have tried to run it witout database name , but still the exception holds. That issue is marked as solved but it doesn't seem so form the comments.
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

anyone?
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

I'm facing the same issue while connecting to my local sandbox. I was told to set a Hive property 'hive.exec.dynamic.partition.mode'  to 'nonstrict', But it didn't help either. I'm using Talend Open Studio for Big Data Version 6.1 
Exception in component tHiveConnection_1

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hello,
I have the same issue in the connection with Open Studio for Big Data 6.1 to Hortonworks 2.3.0 (no sandbox). I tried tHiveConnection and tHiveRow.
Error while processing statement: Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime
In my hive settings, I have also maintained the following Hadoop parameter: mapreduce.application.framework.path=/usr/hdp/2.3.0.0-2557/hadoop/mapreduce.tar.gz#mr-framework
Any idea how to solve it?
Regards,
Moderator

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi cdrweanz,
Could you please upload some job setting screenshots? More information will be preferred.
Best regards
Sabrina
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hello Sabrina,

please find below the job settings:








Except the advanced parameters, which I took out of the Talend 6.1 documentation, I simply took all values from the Cluster Configuration Wizard by connecting to Ambari.
Regards,

 
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi Even am getting same issue. My Hive is running under Tez engine framework and when i run the job getting below issue any solution for this.
Exception in component tHiveConnection_1
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapreduce.framework.name at runtime. It is not in list of params that are allowed to be modified at runtime
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)
Your help will be really appreiciable
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi Jegan,

My Hive is also running under Tez. I found a Jira, which perhaps could help us. Had not yet the time to check it, but it looks promising:
https://jira.talendforge.org/browse/TBD-1504

Regards
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi cdrweanz
Thanks for your reply, i tried to replicate the same whatever the steps which they have mentioned. But does not work out. Kindly share the settings which you have done in your thiveConnection as a screenshot.
Regards,
Jegan
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi Jegan,
I have the same problem. Didn't work. It's still the same message.
Regards
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hello all,

I found the solution. The problem is that if you select from the repository your cluster and keep all standard settings, Talend tries to connect via the data node. For any reasons it's not working. If you change from Repository to Stand Alone and set the following settings it works:

                                                
The advanced settings doesn't need to set as well:


Regards,
                                                                           
 
 
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi cdrweanz,
Thanks a lot, Its working fine now. 
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Hi all, 
I've tried all things i've seen but still not working for me. 
Please, can anyone Help me with this problem and thanks in advance. 
 
 
Employee

Re: Connect Talend to Hive in Hortonworks Sandbox

As cdrweanz mentioned, the workaround is to switch to a built-in property type and uncheck all the Hadoop properties for the tHiveConnection component. See TBD-3260
One Star

Re: Connect Talend to Hive in Hortonworks Sandbox

Screenshots provided by cdrweanz worked for me