topic Re: Set application name using advanced properties in talend spark job in Talend Studio

Set application name using advanced properties in talend spark job

TomG1 — Sat, 16 Nov 2024 07:08:17 GMT

I am trying to change the application name for a spark job created by talend BigData 6.4 version. But this approach is not working. I have seen posts where people change the application name using spark-submit. How do I do the same using talend tool.

Re: Set application name using advanced properties in talend spark job

Anonymous — Tue, 04 Dec 2018 10:48:16 GMT

Hi,

If you are trying to change the name of Talend job, you can close the job and right click the job properties.

Or, are you trying something different?

Warm Regards,

Nikhil Thampi

Re: Set application name using advanced properties in talend spark job

TomG1 — Tue, 04 Dec 2018 11:03:37 GMT

I have attached the screen shot in the question.

Please open that and zoom it.

Dynamic change of talend job name or application name is explained in the following thread.

https://stackoverflow.com/questions/36747268/why-does-conf-setspark-app-name-appname-not-set-the-name-in-the-ui

I want to do the same stuff in talend.

Re: Set application name using advanced properties in talend spark job

TheMadse — Tue, 16 Jul 2019 12:58:16 GMT

Hello @TomG ,

did you found a solution for your issue?
as I have one generic spark job, which is called as child job I would like to set Spark appname during runtime. so it would be possible to determine each execution in in spark history.

Talend Spark Job, sets AppName according sparkcontext as followed

  sparkConfiguration.setAppName(projectName + "_" + jobName + "_"
+ jobVersion);

as I create a sparkSession in java-component I thought to test .appName()

SparkSession.builder().appName("newAppName").getOrCreate();

have you tried this approach?

Re: Set application name using advanced properties in talend spark job

TheMadse — Tue, 16 Jul 2019 13:13:11 GMT

SparkSession.builder().appName("newAppName").getOrCreate();

dose not work

Re: Set application name using advanced properties in talend spark job

Anonymous — Wed, 31 Jul 2019 07:42:10 GMT

Try using "spark.app.name" property in the spark job-> Run-> Spark configuration-> Advanced Settings with the value you would like to set for the job.