topic Re: Unable to build spark standalone Job ( Custom -Unsupported distribution ) in Talend Studio

Unable to build spark standalone Job ( Custom -Unsupported distribution )

Naveenkumar3 — Sat, 16 Nov 2024 01:38:22 GMT

Hi Team,

I am unable to build a spark standalone job and getting the below error message. Can anyone help me to configure spark standalone cluster details in talend and help me to fix this issue.

spark configuration details:

Error message when building the job.

Re: Unable to build spark standalone Job ( Custom -Unsupported distribution )

manodwhb — Wed, 19 Aug 2020 13:15:42 GMT

@Naveenkumar Murugesan , you have some configuration issue in some component. Do you have tFileInput component in your job ? if possible can you show job?

Re: Unable to build spark standalone Job ( Custom -Unsupported distribution )

Naveenkumar3 — Thu, 20 Aug 2020 07:57:56 GMT

@Manohar B , Yes I have tFileInput components. please find the below screenshots.

The job is getting build successfully in the" local mode" and the same job is getting failed in the "standalone mode"

tfileInputDelimited:

Re: Unable to build spark standalone Job ( Custom -Unsupported distribution )

manodwhb — Thu, 20 Aug 2020 13:16:21 GMT

@Naveenkumar Murugesan , you need to check define a storage configuration component and you need to use S3connection or HDFSconnection component and you need to read from the file system.

Thanks,

Manohar

Re: Unable to build spark standalone Job ( Custom -Unsupported distribution )

Naveenkumar3 — Thu, 20 Aug 2020 13:37:59 GMT

@Manohar B We don't have any Hadoop cluster or distributed file system like (S3or HDFS). We are using spark open source standalone cluster and the file system is NFS ( Network file system).

I have mentioned distribution as "Custom unsupported". Do you have any idea what type of .zip file needs to be uploaded in the custom unsupported wizard.

Re: Unable to build spark standalone Job ( Custom -Unsupported distribution )

manodwhb — Thu, 20 Aug 2020 13:45:04 GMT

@Naveenkumar Murugesan , Have you looked at the below link? if not look into it

https://help.talend.com/reader/Ey7t5jMFEKou4HLney5XcQ/x9D~iohZ3_lbTIoDcJDOaQ

Thanks,

Manohar