topic Re: How can I set mapreduce/hive job parameters in a tHiveRow query? in Talend Studio

How can I set mapreduce/hive job parameters in a tHiveRow query?

Anonymous — Wed, 24 Oct 2012 22:06:05 GMT

I am trying to run the following query to insert a new partition into a hive table, but I want to make sure that compression is turned on and that bucketing is envorced. To do that I need to set some job parameters. The following query will not execute if I have the SET param=value lines:
"SET hive.exec.compress.output=true;
SET io.seqfile.compression.type=BLOCK;
SET mapred.output.compression.type=RECORD;
SET mapred.compress.map.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec;
SET hive.enforce.bucketing = true;
INSERT OVERWRITE TABLE t_mytable PARTITION(activity_date='" + TalendDate.formatDate("yyyy-MM-dd", context.DATE) + "') SELECT * from tmp_mytable"
If I remove the SET lines, it works, but does not compress my data. Is there a way to set these parameters? Or do I just have to set them on the hiveserver(thrift) side?

Re: How can I set mapreduce/hive job parameters in a tHiveRow query?

Anonymous — Fri, 26 Oct 2012 07:57:56 GMT

hi
In the SET commands that u have mentiones here, i see the semicolon been put at the end.. (";").Please remove them while being used in tHiveRow.Then it will work.
"SET hive.exec.compress.output=true"
"SET io.seqfile.compression.type=BLOCK"

Re: How can I set mapreduce/hive job parameters in a tHiveRow query?

Anonymous — Wed, 24 Jul 2013 12:31:53 GMT

Hi,
i have a similar issue.
i get my job running in that way :
"SET hive.exec.compress.output=true" +
"SET io.seqfile.compression.type=BLOCK" +
"insert overwrite ..."
or in that way:
"SET hive.exec.compress.output=true;
"SET io.seqfile.compression.type=BLOCK;
insert overwrite ..."
but in both cases the insert statement is not executed. it seems that the first SET statement would be executed and that s all.
Any advice would be good (Talend 5.2).
Jean-Charles