How to exchange flow between child jobs ?

Six Stars

How to exchange flow between child jobs ?

Hi, I'm new to ETL and Talend. I want to separate the following job into two different child jobs managed by a parent. So far, I am able to read the buffer from the first child and output it in a tLogRow, using a tRunJob in the parent makes this straightforward.

But when it comes to the second part, things are much complicated. The child job 2 is unable to read the Buffer so it needs to be changed to a different component. I've read about tFlowToIterate which can be used to store a flow in global map. Unfortunately, I don't know how to design the child2 and link it correctly with the parent.

Can I get some help ?

Capture.PNG

 

 

 


Accepted Solutions
Sixteen Stars

Re: How to exchange flow between child jobs ?

Would it suit your requirement to have your second child job receive one row at a time instead of the whole flow? If so, you are almost there with the tflowToIterate. You would connect your tRunJob for the second child job to the tFlowToIterate and ensure that you child job has a context variable configured for each column you want to process in the row. Then you would use the tRunJob's Context Param table to add the globalMap values created by your tFlowToIterate.

So, as an example, if you have a row with a String column called "name" and an Integer column called "age" and the row feeding the tFlowToIterate was called "row1", your globalMap expressions would look like below.....

 

((String)globalMap.get("row1.name"))

((Integer)globalMap.get("row1.age"))

 

The above expressions would be used in the Context Param section of the tRunJob, supplying the corresponding context variables inside the job with values.

 

If you want to send your whole flow at once, this is possible but can cause memory issues if you are using large datasets. It also requires a fair amount of Java to create a collection, add your data to the collection, pass it to the job as an Object and then cast it back to the collection and use it inside your child job.

 

Alternatively, and much more simple, you can make use of a database or even a flat file to hold your data between child jobs.

 

I'd start with the first way I suggested


All Replies
Sixteen Stars

Re: How to exchange flow between child jobs ?

Would it suit your requirement to have your second child job receive one row at a time instead of the whole flow? If so, you are almost there with the tflowToIterate. You would connect your tRunJob for the second child job to the tFlowToIterate and ensure that you child job has a context variable configured for each column you want to process in the row. Then you would use the tRunJob's Context Param table to add the globalMap values created by your tFlowToIterate.

So, as an example, if you have a row with a String column called "name" and an Integer column called "age" and the row feeding the tFlowToIterate was called "row1", your globalMap expressions would look like below.....

 

((String)globalMap.get("row1.name"))

((Integer)globalMap.get("row1.age"))

 

The above expressions would be used in the Context Param section of the tRunJob, supplying the corresponding context variables inside the job with values.

 

If you want to send your whole flow at once, this is possible but can cause memory issues if you are using large datasets. It also requires a fair amount of Java to create a collection, add your data to the collection, pass it to the job as an Object and then cast it back to the collection and use it inside your child job.

 

Alternatively, and much more simple, you can make use of a database or even a flat file to hold your data between child jobs.

 

I'd start with the first way I suggested

Six Stars

Re: How to exchange flow between child jobs ?

Much appreciated, Sir.
Solved it using context params in tRunJob