topic Talend Big Data: HIVE Table tHiveLoad OVERWRITE doesn´t work in Talend Studio

Talend Big Data: HIVE Table tHiveLoad OVERWRITE doesn´t work

Anonymous — Sat, 16 Nov 2024 11:41:35 GMT

Hi guys,
I need to run a job in which I have to truncate every time an Hive Table and insert new data.
If I execute the query from the HUE browser interface using the same user like this:
INSERT OVERWRITE TABLE work.bi_xxx
SELECT ...
the map reduce job works properly. :-)
If I translate this into a Talend Job using a tHiveLoad and using:
- the SAME user
- and Load action: INSERT
- Action on file: OVERWRITE
it doesn´t work and I receive this exception: :-(
Default;6;Java Exception;tHiveLoad_2;java.sql.SQLException:Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask;1

If I execute the same but with Action on File: APPEND it works properly. :-)
How can be considering that I´m using the same user with the same permissions?
I tried also to use a tHiveRow component before in order to truncate the table and then using APPEND in the tHiveLoad but the TRUNCATE command doesn´t work.
Please can you help me?
I really don´t know why this different behavior.
Thanks in advance
Mary

Re: Talend Big Data: HIVE Table tHiveLoad OVERWRITE doesn´t work

willm1 — Mon, 07 Apr 2014 15:03:35 GMT

Mary - does the APPEND work in HUE browser or in Talend using the Hive component?

Re: Talend Big Data: HIVE Table tHiveLoad OVERWRITE doesn´t work

Anonymous — Mon, 07 Apr 2014 18:02:31 GMT

HI,
yes, the APPEND in Talend works properly but not the OVERWRITE.
Do you have any idea?
Thanks in advance
Mary

Re: Talend Big Data: HIVE Table tHiveLoad OVERWRITE doesn´t work

Anonymous — Tue, 22 Jul 2014 20:02:12 GMT

Does the underlying HDFS files (or partitions) have necessary permissions for the Talend user. Try giving 777 to the underlying HDFS file. Also grant all on the table to the user running the job.
I am using 5.4 big data platform and seems to run fine. The Hadoop distribution is Horton works HDP V1.2.
The SQL from Talend is:
INSERT OVERWRITE TABLE EDW_SHRD_VIEWS.DIM_DELTA SELECT NW.nm , NW.attr , NW.bgn_dt , NW.id , NW.end_dt , CASE WHEN OD.ID IS NULL THEN 'I' WHEN OD.ID IS NOT NULL AND OD.NM != NW.NM THEN 'UI' WHEN OD.ID is NOT NULL AND OD.ATTR !=NW.ATTR THEN 'U' ELSE 'N' END AS CHG_IND, OD.nm , OD.attr , OD.bgn_dt , OD.id , OD.end_dt FROM EDW_SHRD_VIEWS.STG_DIM NW LEFT OUTER JOIN EDW_SHRD_VIEWS.DIM OD ON( OD.id = NW.id )