Execute Spark batch job as root.user

Highlighted
Six Stars

Execute Spark batch job as root.user

Hi, I have a problem running a Spark batch job on MapR.

 

I'm currently working on a Spark batch job on a cluster with multiple nodes.

But when I ran the job, the job fails, with this message as an output:

"Yarn application has already ended! It might have been killed or unable to launch application master."

 

I checked the resource manager log and saw that the job failed with this message:

"Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500."

 

I think the job fails when it runs as user "root". So is there any way I can run the job as a user besides root?

Moderator

Re: Execute Spark batch job as root.user

Hello,

We have a jira issue:https://jira.talendforge.org/browse/TBD-3955 about this error: "Requested user root is not whitelisted and has id 0,which is below the minimum allowed 500".

Let us know if the solution works for you.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi Sabrina,

 

Thanks for responding to the problem.

 

If I understand correctly, the solution to the Jira issue was to use a JobServer.

But I can't find a JobServer directory in my Talend installation directory. I've only managed to find several jars which relate to JobServer.

 

From this link (https://help.talend.com/reader/ozfCR4oo4noN8gSKT7qP_w/TgjvBs4NXECwoSePsMB0HQ), it seems that there should be a folder named JobServer.

 

Do I have to do a separate installation in order to use it? Or is the JobServer directory stored somewhere else?

Moderator

Re: Execute Spark batch job as root.user

Hello,

Are you using talend data platform product? Free trial version? Is there any archive file name for example reads: Talend-JobServer-YYYYMMDD_HHmm-VA.B.C.zip?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi,

 

I'm using a licensed version of Talend Data Fabric.

But no, I can't find a zip file under that name.

Moderator

Re: Execute Spark batch job as root.user

Hi,

With you licensed version of Talend Data Fabric, could you please create a case on talend support portal so that we can give you a remote assistance(webex) to see if there is any problem for your job server installation through support cycle with priority?

https://login.talend.com/support-login.php.

Best regards

Sabrina

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Execute Spark batch job as root.user

Hi,

 

Okay, will try to contact the support team. Thank you Sabrina.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now