topic How can I set mapreduce/hive job parameters in a tHiveInput query? in Talend Studio

How can I set mapreduce/hive job parameters in a tHiveInput query?

Anonymous — Sat, 16 Nov 2024 09:42:55 GMT

I am trying to export the output of the following query . Before that I want to set some hive properties.

SET hive.execution.engine=tez;

SET hive.tez.container.size=8192;

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

set hive.merge.tezfiles=true;

I am building the query using various context variables.

"select tbl.hd from (select tab1.hd, tab1.cl1 from (" + context.getProperty("CUSTOM_SEL") + context.getProperty("HDR_COL) + context.getProperty("CUST_HEADER") + context.getProperty("CUST_UNION") + context.SQL_TEXT+") tab1 order by tab1.cl1 asc ) tbl"

Is there way I can execute the SET hive parameters before executing select query ?

Re: How can I set mapreduce/hive job parameters in a tHiveInput query?

Anonymous — Thu, 01 Jun 2017 18:53:19 GMT

In the tHiveConnection component, click the 'Advanced settings', click the '+' under 'Hive Properties' and for each property:

Property

"hive.execution.engine"

Value

"tez"

Re: How can I set mapreduce/hive job parameters in a tHiveInput query?

Anonymous — Fri, 30 Jun 2017 23:53:46 GMT

Yes. That works. But we have parameterized the query and we want to parameterize those "HIVE Set params" through a context variable. That way we can set different set of parameters for different hive queries. This way we have to hard code it for all the queries.