Talend Connect
Virtual Summit
JOIN US!
And visit the Customer
& Community Lounge.
May 27-28, wherever you are.

how to pass data from one subjob to other sub job

Highlighted
Four Stars

how to pass data from one subjob to other sub job

Hi,

 

I have a subjob1 where I read a file and do simple tlogrow. 

I have linked this subjob1 to subjob2 through component onsubjobok.

my tlogrow display correct values for the input file. But these values are never passed in subjob2.

I'm using big data spark jobs. how do i pass the value from subjob1 to subjob2..?

Highlighted
Forteen Stars

Re: how to pass data from one subjob to other sub job

tSubJobOk - do not transfer data

 

in your case - you are "read and forget"

 

You can connect tlogrow (Main) to next component or read file again in proper place

-----------
Highlighted
Four Stars

Re: how to pass data from one subjob to other sub job

@vapukov Thanks for the reply.

 

when i right click on tlogrow, it doesnt give me any option on how to connect to the next job.

My Job1 is : tfileinputparquet -> tmap1-> tlogrow (in the tmap here, i have hashmap function). 

The output of the tmap1 needs to be passed to job2..

Highlighted
Forteen Stars

Re: how to pass data from one subjob to other sub job

You can not pass result from job to job even in data integration and in bigdata as well

 

if You want use result of first job in next - save result in file (or etc)

if You use main connection - it always will be same job

 

but if in your case tLog it only Log, You can parallel it with file output

tLog.PNG

tlog2.PNG

-----------
Highlighted
Four Stars

Re: how to pass data from one subjob to other sub job

Thanks @vapukov, i got your point.

However in my case the scenario is different.  If i output the result tfileparquetoutput, then i have to join my subjob2 input to tfileparquetoutput. However i dont have any join key in common.

So i wanted to pass the value through object by creating tjava component b/w the 2 jobs. but that doesnt work

 

Highlighted
Forteen Stars

Re: how to pass data from one subjob to other sub job

I think - You are not got my points:

 

what reason of tLogrow in real job? do You want print on screen (or in TAC log files) 1-2-3-10M rows of text?

remove and connect output from tmap to proper component

 

also, You always can add key to output file, for example - row number ( sequenced )

-----------

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now