Six Stars

Return Values from a child Spark job to parent DI job

Hi,
I want to retrieve values back from Spark child job to DI parent as we do for DI parent and child.
I have followed below link to fetch value from Spark child job.
http://www.talendbyexample.com/talend-returning-values-from-subjobs.html
When I unselect "Use an independent process to run subjob" property in tRunJob it is not working. Since we need to run Spark jobs independently through which Jobs can't communicate each other.
Below is the topic I worked to implement with DI jobs.
https://www.talendforge.org/forum/viewtopic.php?id=54045
Thanks and appreciate your help,
L Raghunath.
Please see below screen shot for message shown when I uncheck "Use an independent process to run subjob" property in tRunJob for Spark child job.
Raghunath Lingutla
1 REPLY
Six Stars JR
Six Stars

Re: Return Values from a child Spark job to parent DI job

Hi Raghunath
Have you tried to store the value in a file in your child job (some sort of context dump) and then read the contents of the file in your parent job (context load)? I am not familiar with Spark jobs but is seems to me as if you will have to completely decouple both jobs and use an intermediate storage that is not residing in the memory.
Regards, Joachim