topic Erorr SparkTask in Talend Studio

Erorr SparkTask

Anonymous — Sat, 16 Nov 2024 06:39:37 GMT

Hello everyone

I am using Talend Big Data Platform to implement ELT Jobs with Hive tables. The jobs run with Spark engine.

I have a problem with a Hive on Spark job.

The scope of the task is to join two tables (parquet) and then insert the output into another table (parquet).

The job goes wrong but I am not able to interpet it.

This is the error message:

[FATAL]: datahub_rp_cli.j_rp_cli_movimenti_0_1.J_RP_CLI_MOVIMENTI - tRunJob_2 Child job running failed
[statistics] disconnected
Exception in component tELTHiveOutput_9
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:279)

Trying to find the solution online in the official documentation or forum, I understand that the errors SparkTask (with code 1 or 2 or 3 eccetera) are something like a 'wrapper' of java errors; however I cannot access to hive log because of denied permissions on the server.

So have you got a lists of causes that can originate somekind of java errors for Sparktask error code =3 ?

Thank you.

Re: Erorr SparkTask

Anonymous — Thu, 28 Feb 2019 07:09:54 GMT

Hello,

Could you please make sure that all required external jars in your job are installed successfully in your studio? On which talend bigdata platform build version you got this issue?

Best regards

Sabrina

Re: Erorr SparkTask

Anonymous — Thu, 28 Feb 2019 08:39:00 GMT

Hi @xdshi

I work with Talend Big Data Platform 6.3.1.

Where may I check if in Talend I have got all the right external Jars exactly installed?

However I have succeeded in resolving the problem in this way:

The Hive ELT process got error SparkTask at the table that is partictioned daily from 2011 to 2018; I tested the Talend ELT job for a subset of this partitioned table and then it finished the execution well, without sparkTask error.

I imagine that I have to compute statistic for that table on hive, and so It is not a Talend software issue.

Thank you

Re: Erorr SparkTask

Anonymous — Thu, 28 Feb 2019 09:25:08 GMT

Hello,

Great you could work on your solution.

Here is online documentation about:TalendHelpCenter:Installing external modules

Best regards

Sabrina