How can I set mapreduce/hive job parameters in a tHiveInput query?

Four Stars

How can I set mapreduce/hive job parameters in a tHiveInput query?

I am trying to export the output of the following query . Before that I want to set some hive properties.

SET hive.execution.engine=tez;

SET hive.tez.container.size=8192;

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

set hive.merge.tezfiles=true;

 

I am building the query using various context variables.

"select tbl.hd from (select tab1.hd, tab1.cl1 from (" + context.getProperty("CUSTOM_SEL") + context.getProperty("HDR_COL) + context.getProperty("CUST_HEADER") + context.getProperty("CUST_UNION") + context.SQL_TEXT+") tab1 order by tab1.cl1 asc ) tbl"

Is there way I can execute the SET hive parameters before executing select query ?

Seven Stars

Re: How can I set mapreduce/hive job parameters in a tHiveInput query?

In the tHiveConnection component, click the 'Advanced settings', click the '+' under 'Hive Properties' and for each property:

Property

"hive.execution.engine"

 

Value

"tez"

Four Stars

Re: How can I set mapreduce/hive job parameters in a tHiveInput query?

Yes. That works. But we have parameterized the query and we want to parameterize those   "HIVE Set params" through a context variable. That way we can set different set of parameters for different hive queries. This way we have to hard code it for all the queries.

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now