Five Stars

How to set and use globalVariable/context variable in Big Data Batch job

Hi All,

 

I am not able to set globalVariable/context variable in Big Data Batch job.  While, I can do so in standard job.

 

Is there any other way to set and use globalVariable/context?

Tags (2)
2 REPLIES
Moderator

Re: How to set and use globalVariable/context variable in Big Data Batch job

Hello,

For bigdata batch job, the talend recommendation is to use a DI job 'launcher' that loads the "dynamic" context.

So far, there is also no tContextLoad component for Big Data Batch and Spark Streaming.

Here is a related article about:https://community.talend.com/t5/Architecture-Best-Practices-and/Spark-Dynamic-Context/ta-p/33038/jum...

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: How to set and use globalVariable/context variable in Big Data Batch job

Thanks Sabrina.

We were able to set context variable from DI job and pass it to Spark job.

But we are not able to reset that variable during runtime in spark job.

For e.g. if we want to capture status of certain operation in spark job, we
are not able to set variable as "success" or "failed".