Spark Jobs in Talend Big Data

[nnmh]

I have a problem running any Big Data Batch job in Spark configuration which that any job takes a long time to deploy and start the job.

Every time I started a Big Data Batch job it uploads the job jars to HDFS dir which takes a long time.

Are there any spark configuration is required to avoid uploading these jars every time the job starts?

[Anonymous]

Hello @Moataz Nader​ ,

What's the Spark Mode in your job's RUN/Spark Configuration?

How about to change it to 'Yarn Cluster' to see if the performance is better?

 

Best regards

Aiming

[nnmh]

Dear @Aiming Chen​ 

Spark Mode in my job's RUN/Spark Configuration

Every time it runs it take a lot of time to upload its jars on HDFS