How to exchange flow between child jobs ?

Eight Stars

How to exchange flow between child jobs ?

Hi, I'm new to ETL and Talend. I want to separate the following job into two different child jobs managed by a parent. So far, I am able to read the buffer from the first child and output it in a tLogRow, using a tRunJob in the parent makes this straightforward.

But when it comes to the second part, things are much complicated. The child job 2 is unable to read the Buffer so it needs to be changed to a different component. I've read about tFlowToIterate which can be used to store a flow in global map. Unfortunately, I don't know how to design the child2 and link it correctly with the parent.

Can I get some help ?

Capture.PNG

 

 

 


Accepted Solutions
Community Manager

Re: How to exchange flow between child jobs ?

Would it suit your requirement to have your second child job receive one row at a time instead of the whole flow? If so, you are almost there with the tflowToIterate. You would connect your tRunJob for the second child job to the tFlowToIterate and ensure that you child job has a context variable configured for each column you want to process in the row. Then you would use the tRunJob's Context Param table to add the globalMap values created by your tFlowToIterate.

So, as an example, if you have a row with a String column called "name" and an Integer column called "age" and the row feeding the tFlowToIterate was called "row1", your globalMap expressions would look like below.....

 

((String)globalMap.get("row1.name"))

((Integer)globalMap.get("row1.age"))

 

The above expressions would be used in the Context Param section of the tRunJob, supplying the corresponding context variables inside the job with values.

 

If you want to send your whole flow at once, this is possible but can cause memory issues if you are using large datasets. It also requires a fair amount of Java to create a collection, add your data to the collection, pass it to the job as an Object and then cast it back to the collection and use it inside your child job.

 

Alternatively, and much more simple, you can make use of a database or even a flat file to hold your data between child jobs.

 

I'd start with the first way I suggested


All Replies
Community Manager

Re: How to exchange flow between child jobs ?

Would it suit your requirement to have your second child job receive one row at a time instead of the whole flow? If so, you are almost there with the tflowToIterate. You would connect your tRunJob for the second child job to the tFlowToIterate and ensure that you child job has a context variable configured for each column you want to process in the row. Then you would use the tRunJob's Context Param table to add the globalMap values created by your tFlowToIterate.

So, as an example, if you have a row with a String column called "name" and an Integer column called "age" and the row feeding the tFlowToIterate was called "row1", your globalMap expressions would look like below.....

 

((String)globalMap.get("row1.name"))

((Integer)globalMap.get("row1.age"))

 

The above expressions would be used in the Context Param section of the tRunJob, supplying the corresponding context variables inside the job with values.

 

If you want to send your whole flow at once, this is possible but can cause memory issues if you are using large datasets. It also requires a fair amount of Java to create a collection, add your data to the collection, pass it to the job as an Object and then cast it back to the collection and use it inside your child job.

 

Alternatively, and much more simple, you can make use of a database or even a flat file to hold your data between child jobs.

 

I'd start with the first way I suggested

Eight Stars

Re: How to exchange flow between child jobs ?

Much appreciated, Sir.
Solved it using context params in tRunJob

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download