topic Re: Talend spark job in cluster mode in Talend Studio

Talend spark job in cluster mode

Anonymous — Sat, 16 Nov 2024 10:14:36 GMT

How can talend spark job be executed in cluster mode? Talend supports local, stand alone and yarn-client mode. I want to run it in yarn cluster mode. Is it possible?

Re: Talend spark job in cluster mode

Anonymous — Thu, 24 Nov 2016 06:40:58 GMT

Hi,
We have redirected your issue to our bigdata experts and then come back you as soon as we can.
Best regards
Sabrina

Re: Talend spark job in cluster mode

Anonymous — Thu, 24 Nov 2016 08:41:06 GMT

Hi,
The yarn-cluster mode is currently not supported in the studio.
Regards,
Rémy.

Re: Talend spark job in cluster mode

Anonymous — Fri, 25 Nov 2016 04:09:08 GMT

Is there any way I can take the export, change the batch file / shell file to achieve it? I know that we can pass deploy mode (yarn-client vs yarn-cluster) as an argument to the spark-submit shell command.
Can the export shell file be modified to achieve the same?

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \

Re: Talend spark job in cluster mode

Anonymous — Fri, 07 Sep 2018 20:57:55 GMT

Hi Sankalp,

Did you find a way to run Talend job jar over a cluster by spark-submit ?