Five Stars

How to share data between parent job and child jobs.

I want to save data in java hashtable or something similar, but seems like I cannot save it in globalmap and reuse it in the other subjob. Is there any feature in Talend I can use to do it?

 

BTW, the object been passed needs to be thread safe.

 

 

  • Data Integration
6 REPLIES
Ten Stars

Re: How to share data between parent job and child jobs.

You can do this using context variables. However it is not straight forward unless you are familiar with Java. If you are familiar with Java, these instructions should open the door to you....

 

1) Lets assume you want to send an ArrayList of Strings to your child job. Use a tJavaFlex to create the ArrayList and populate it. In the "End Code" section of the tJavaFlex, assign the ArrayList to the globalMap.

2) Connect (using an OnComponentOK or OnSubJobOK) to a tRunJob. Go to the context config section and set the value of one of the contexts to the ArrayList stored in your globalMap. The context you set it to MUST be an Object

3) Inside your child job, retrieve your ArrayList from the context variable you passed it to (maybe using a tJavaFlex). You MUST cast it to an ArrayList before use.

 

This way you can send a collection of values to your child job. If you want to send a row of data, then the best way to do this is to create a data class (in a routine) and assign each row's data to the instance of that class. Then assign that instance to your ArrayList (then as above).

 

I use a mechanism like this in this tutorial (https://www.rilhia.com/tutorials/talend-connect-example).

 

I have also written a component to handle this for you which is located here (https://exchange.talend.com/#vproductitemoverview:vendor=orgs%252F1%252Fvendors%252F2034&p=orgs%252F...)

Rilhia Solutions
Five Stars

Re: How to share data between parent job and child jobs.

Hi,

 

Thank you very much for your reply. Actually I have tried this method. The only reason it does not work for me is it can not send the data back to the parent job. Do you know if there is anyway we can set up communication between parent and child jobs for sharing java objects? 

Ten Stars

Re: How to share data between parent job and child jobs.

Five Stars

Re: How to share data between parent job and child jobs.

Is there a way we can create something in routine to hold data throughout the whole job including its subjobs?

Ten Stars

Re: How to share data between parent job and child jobs.

It might be easier if you tell me what exactly you want to do. What you have asked is possible.....but you may be sacrificing supportability (if that is a word) to do something largely in Java. While one of the advantages of Talend is that you can build upon it using Java, you have to be very careful not to "reinvent the wheel" in order to do something in a particular way when an alternative method has already been implemented and is supported by Talend. 

 

The original question you raised was can you send a collection of data to a subjob. Yes you can, and there are several use cases I can think for that. Can it be delivered back in the same format? Yes. You can return a single row with a single Object column using a tBufferOutput, and send the same sort of data back to the parent (where you would have to cast it to ArrayList again, etc). But using this structure means that you will have to waste processor cycles packaging it up and extracting it every time you want to do anything with it using standard Talend components. In the end you will likely build routines to process your package data structure and then you end up with Talend acting as an orchestrator of your bespoke code.....which is not ideal.

 

There is always room for a bit of bespoke Java, but it should be seriously considered as to whether it is actually needed. 

Rilhia Solutions
Moderator

Re: How to share data between parent job and child jobs.

Hi,

Here are troubleshooting-development posts about:https://community.talend.com/t5/Troubleshooting-Development/Passing-a-value-from-a-parent-Job-to-a-c... and https://community.talend.com/t5/Troubleshooting-Development/Returning-a-value-from-a-child-Job-to-th...

Let us know if it is what you are looking for.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.