I am using Talend Big Data Platform to implement ELT Jobs with Hive tables. The jobs run with Spark engine.
I have a problem with a Hive on Spark job.
The scope of the task is to join two tables (parquet) and then insert the output into another table (parquet).
The job goes wrong but I am not able to interpet it.
This is the error message:
[FATAL]: datahub_rp_cli.j_rp_cli_movimenti_0_1.J_RP_CLI_MOVIMENTI - tRunJob_2 Child job running failed [statistics] disconnected Exception in component tELTHiveOutput_9 java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:279)
Trying to find the solution online in the official documentation or forum, I understand that the errors SparkTask (with code 1 or 2 or 3 eccetera) are something like a 'wrapper' of java errors; however I cannot access to hive log because of denied permissions on the server.
So have you got a lists of causes that can originate somekind of java errors for Sparktask error code =3 ?
Could you please make sure that all required external jars in your job are installed successfully in your studio? On which talend bigdata platform build version you got this issue?
Where may I check if in Talend I have got all the right external Jars exactly installed?
However I have succeeded in resolving the problem in this way:
The Hive ELT process got error SparkTask at the table that is partictioned daily from 2011 to 2018; I tested the Talend ELT job for a subset of this partitioned table and then it finished the execution well, without sparkTask error.
I imagine that I have to compute statistic for that table on hive, and so It is not a Talend software issue.
