Set application name using advanced properties in talend spark job

Highlighted
Seven Stars

Set application name using advanced properties in talend spark job

I am trying to change the application name for a spark job created by talend BigData 6.4 version. But this approach is not working. I have seen posts where people change the application name using spark-submit. How do I do the same using talend tool.

 

 

Employee

Re: Set application name using advanced properties in talend spark job

Hi,

 

    If you are trying to change the name of Talend job, you can close the job and right click the job properties.

image.png

 

 

Or, are you trying something different?

 

Warm Regards,

 

Nikhil Thampi

Seven Stars

Re: Set application name using advanced properties in talend spark job

I have attached the screen shot in the question.

Please open that and zoom it.

 

Dynamic change of talend job name or application name is explained in the following thread.

 

https://stackoverflow.com/questions/36747268/why-does-conf-setspark-app-name-appname-not-set-the-nam...

 

 

I want to do the same stuff in talend.

Four Stars

Re: Set application name using advanced properties in talend spark job

Hello @TomG ,

did you found a solution for your issue?
as I have one generic spark job, which is called as child job I would like to set Spark appname during runtime. so it would be possible to determine each execution in in spark history.

 

Talend Spark Job, sets AppName according sparkcontext as followed

  sparkConfiguration.setAppName(projectName + "_" + jobName + "_"
+ jobVersion);


  as I create a sparkSession in java-component I thought to test .appName()

SparkSession.builder().appName("newAppName").getOrCreate();

have you tried this approach?

Four Stars

Re: Set application name using advanced properties in talend spark job

SparkSession.builder().appName("newAppName").getOrCreate();

dose not work Smiley Sad

Employee

Re: Set application name using advanced properties in talend spark job

Try using "spark.app.name" property in the spark job-> Run-> Spark configuration-> Advanced Settings with the value you would like to set for the job.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now