How to pass a data flow to a subjob in TOS?

One Star

How to pass a data flow to a subjob in TOS?

Hi Team,
I am trying to build a Talend project which has PARENT and CHILD job design. In parent job i plan to continuously read json messages from tKafkaInput component and then replicate this data into different branches. And each branch will extract json message as per the DB table schema and then call corresponding table child job.
JSON message schema: It has metadata columns like tablename, timestamp, and database table1 columns and like wise other json message will have tablename, timestamp, and database table2 columns in schema
Problem: How to pass continuous stream of data (whole dataset) to child job? And how to read this data flow(dataset) in child job for further processing?
Thanks for your help and time in advance.
Re
Community Manager

Re: How to pass a data flow to a subjob in TOS?

Hi  
You can pass the message to the child job and read the message with tExtractJsonFields, please refers to this KB article to learn how to pass a value from parent job to child job. 
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: How to pass a data flow to a subjob in TOS?

Thank You, Shong for your reply.
I am able to pass JSON message from parent to child job by making use of context variables but the challenge is how do i use tExtractJson component in the child job without having any inputs?
Please suggest how to use json message stored in a context variable in a child job to further extract it into columns and then process it?
Thanks so much!!
Rera
Community Manager

Re: How to pass a data flow to a subjob in TOS?

You can use a tFixedFlowInput to generate the message, to do it: add a column and set its value with context variable. 
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: How to pass a data flow to a subjob in TOS?

That works but it process just one row at a time. In my flow i can receive 1000s of rows in few ms in that case how do i acheive processing multiple rows at a time in the child job.
Thanks for your help!
One Star

Re: How to pass a data flow to a subjob in TOS?

Hi Shong/Team,
Using tFixedFlowInput to pass data flow from PARENT to CHILD job results in sending one row at a time to child job. This will effect job performance because it is very likely that we will receive continuous stream of JSON message from Kafka and this can push 1000s of rows in a millisecond. So, that is why i am looking for a approach using which we can pass whole dataset in one go to child job for processing instead of a single row.
Please let me know how can i achieve this?
Thanks in advance!
Rera
Community Manager

Re: How to pass a data flow to a subjob in TOS?

Hi Rera
It is impossible to pass all rows from parent job to child at a time, you need to review your job design. Why you need to process the data in a separate child job? In this topic, I have suggested you to read the message from kafka and process the data in child job. 
Regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Sixteen Stars

Re: How to pass a data flow to a subjob in TOS?

You can actually do this, but it is complicated and you have to be careful with memory. I have written a tutorial which uses this type of functionality to emulate an Oracle "Connect-By" analytical function. The whole data set is passed to a child job. This uses a lot of Java. It can be found here (http://rilhia.com/tutorials/talend-connect-example).
A slightly easier way to implement this without so much code is to use the tDataRowToContext component that I created for this very purpose. You can find it in the Exchange. 
Since both methods involve packaging the complete dataset up into a context variable which is passed between the Parent and Child job, you have to be careful about memory. This can be mitigated by breaking the dataset into chunks using a tLoop for example. 
One Star

Re: How to pass a data flow to a subjob in TOS?

Thanks, Chong for your input. I want to process in a separate child job because in Parent job m ONLY ONCE reading/using tkafkainput and then passing the read kafka json message to different child jobs to process as per the message metadata information.
Thank you, Rhall. I will try to use tDataRowToContext component and see if that works out for me.
One Star

Re: How to pass a data flow to a subjob in TOS?

Hi Rhall,
I tried using  tDataRowToContext component in my parent-child job design and i m getting below exception for my child job tDataRowToContext component:
Exception in component tDataRowToContext_1
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.ArrayList
Talend version used is 6.1.1 and i have created context variable of OBJECT data type in both parent and child job. Still m getting an exception. Can you please guide me in fixing this issue?
Thanks for your help!!
Rera