topic Re: Spark Jobs in Talend Big Data in Talend Studio

Spark Jobs in Talend Big Data

nnmh — Fri, 15 Nov 2024 21:45:10 GMT

I have a problem running any Big Data Batch job in Spark configuration which that any job takes a long time to deploy and start the job.

Every time I started a Big Data Batch job it uploads the job jars to HDFS dir which takes a long time.

Are there any spark configuration is required to avoid uploading these jars every time the job starts?

Anonymous — Wed, 31 May 2023 03:15:48 GMT

Hello @Moataz Nader ,

What's the Spark Mode in your job's RUN/Spark Configuration?

How about to change it to 'Yarn Cluster' to see if the performance is better?

Best regards

Aiming

nnmh — Wed, 31 May 2023 11:24:30 GMT

Dear @Aiming Chen

Spark Mode in my job's RUN/Spark Configuration

Every time it runs it take a lot of time to upload its jars on HDFS