Return Values from a child Spark job to parent DI job
Hi, I want to retrieve values back from Spark child job to DI parent as we do for DI parent and child. I have followed below link to fetch value from Spark child job. http://www.talendbyexample.com/talend-returning-values-from-subjobs.html When I unselect "Use an independent process to run subjob" property in tRunJob it is not working. Since we need to run Spark jobs independently through which Jobs can't communicate each other. Below is the topic I worked to implement with DI jobs. https://www.talendforge.org/forum/viewtopic.php?id=54045 Thanks and appreciate your help, L Raghunath. Please see below screen shot for message shown when I uncheck "Use an independent process to run subjob" property in tRunJob for Spark child job.
Re: Return Values from a child Spark job to parent DI job
Hi Raghunath Have you tried to store the value in a file in your child job (some sort of context dump) and then read the contents of the file in your parent job (context load)? I am not familiar with Spark jobs but is seems to me as if you will have to completely decouple both jobs and use an intermediate storage that is not residing in the memory. Regards, Joachim