Six Stars

Execute Spark batch job as root.user

Hi, I have a problem running a Spark batch job on MapR.

 

I'm currently working on a Spark batch job on a cluster with multiple nodes.

But when I ran the job, the job fails, with this message as an output:

"Yarn application has already ended! It might have been killed or unable to launch application master."

 

I checked the resource manager log and saw that the job failed with this message:

"Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500."

 

I think the job fails when it runs as user "root". So is there any way I can run the job as a user besides root?

Tags (3)
6 REPLIES
Moderator

Re: Execute Spark batch job as root.user

Hello,

We have a jira issue:https://jira.talendforge.org/browse/TBD-3955 about this error: "Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500".

Let us know if the solution works for you.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi Sabrina,

 

Thanks for responding to the problem.

 

If I understand correctly, the solution to the Jira issue was to use a JobServer.

But I can't find a JobServer directory in my Talend installation directory. I've only managed to find several jars which relate to JobServer.

 

From this link (https://help.talend.com/reader/ozfCR4oo4noN8gSKT7qP_w/TgjvBs4NXECwoSePsMB0HQ), it seems that there should be a folder named JobServer.

 

Do I have to do a separate installation in order to use it? Or is the JobServer directory stored somewhere else?

Moderator

Re: Execute Spark batch job as root.user

Hello,

Are you using talend data platform product? Free trial version? Is there any archive file name for example reads: Talend-JobServer-YYYYMMDD_HHmm-VA.B.C.zip?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi,

 

I'm using a licensed version of Talend Data Fabric.

But no, I can't find a zip file under that name.

Moderator

Re: Execute Spark batch job as root.user

Hi,

With you licensed version of Talend Data Fabric, could you please create a case on talend support portal so that we can give you a remote assistance(webex) to see if there is any problem for your job server installation through support cycle with priority?

https://login.talend.com/support-login.php.

Best regards

Sabrina

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi,

 

Okay, will try to contact the support team. Thank you Sabrina.