topic Re: Execute Spark batch job as root.user in Talend Studio

Execute Spark batch job as root.user

Anonymous — Sat, 16 Nov 2024 09:17:16 GMT

Hi, I have a problem running a Spark batch job on MapR.

I'm currently working on a Spark batch job on a cluster with multiple nodes.

But when I ran the job, the job fails, with this message as an output:

"Yarn application has already ended! It might have been killed or unable to launch application master."

I checked the resource manager log and saw that the job failed with this message:

"Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500."

I think the job fails when it runs as user "root". So is there any way I can run the job as a user besides root?

Re: Execute Spark batch job as root.user

Anonymous — Mon, 18 Sep 2017 07:00:12 GMT

Hello,

We have a jira issue:https://jira.talendforge.org/browse/TBD-3955 about this error: "Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500".

Let us know if the solution works for you.

Best regards

Sabrina

Re: Execute Spark batch job as root.user

Anonymous — Mon, 18 Sep 2017 07:43:11 GMT

Hi Sabrina,

Thanks for responding to the problem.

If I understand correctly, the solution to the Jira issue was to use a JobServer.

But I can't find a JobServer directory in my Talend installation directory. I've only managed to find several jars which relate to JobServer.

From this link (https://help.talend.com/reader/ozfCR4oo4noN8gSKT7qP_w/TgjvBs4NXECwoSePsMB0HQ), it seems that there should be a folder named JobServer.

Do I have to do a separate installation in order to use it? Or is the JobServer directory stored somewhere else?

Re: Execute Spark batch job as root.user

Anonymous — Mon, 18 Sep 2017 08:32:04 GMT

Hello,

Are you using talend data platform product? Free trial version? Is there any archive file name for example reads: Talend-JobServer-YYYYMMDD_HHmm-VA.B.C.zip?

Best regards

Sabrina

Re: Execute Spark batch job as root.user

Anonymous — Mon, 18 Sep 2017 08:38:20 GMT

Hi,

I'm using a licensed version of Talend Data Fabric.

But no, I can't find a zip file under that name.

Re: Execute Spark batch job as root.user

Anonymous — Mon, 18 Sep 2017 08:47:10 GMT

Hi,

With you licensed version of Talend Data Fabric, could you please create a case on talend support portal so that we can give you a remote assistance(webex) to see if there is any problem for your job server installation through support cycle with priority?

https://login.talend.com/support-login.php.

Best regards

Sabrina

Re: Execute Spark batch job as root.user

Anonymous — Wed, 20 Sep 2017 08:47:45 GMT

Hi,

Okay, will try to contact the support team. Thank you Sabrina.