Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
I have a problem running any Big Data Batch job in Spark configuration which that any job takes a long time to deploy and start the job.
Every time I started a Big Data Batch job it uploads the job jars to HDFS dir which takes a long time.
Are there any spark configuration is required to avoid uploading these jars every time the job starts?
Hello @Moataz Nader ,
What's the Spark Mode in your job's RUN/Spark Configuration?
How about to change it to 'Yarn Cluster' to see if the performance is better?
Best regards
Aiming
Dear @Aiming Chen
Spark Mode in my job's RUN/Spark Configuration
Every time it runs it take a lot of time to upload its jars on HDFS