Talend Connect
Virtual Summit
JOIN US!
And visit the Customer
& Community Lounge.
May 27-28, wherever you are.

Merge output of two job in a single output file

Highlighted
Six Stars

Merge output of two job in a single output file

Hi All,

How to merge two job output in a single output file.I am adding screenshot of my job:

JobScreenshotpicture.png

I have to merge tFileOutputDelimited_3 and tFileOutputDelimited_4 out put in a single output file.Thanks in advance. 

 

 


Accepted Solutions
Highlighted
Forteen Stars

Re: Merge output of two job in a single output file

because You are attaching as a source my example (sure not the best way, but it was created fast on one of the projects)

 

You can see below on the same page - for convert csv (comma delimited) string, I read from a file as \t (TAB) delimited, in this case, all row it a single column

 

If You check other links, You can create more elegant code of course

 

P.S.

You always could use csvkit, I use it as well from talend for different reasons

-----------

View solution in original post

Highlighted
Six Stars

Re: Merge output of two job in a single output file

Solved using Jackson library, by creating a routine in talend.

View solution in original post


All Replies
Highlighted
Forteen Stars

Re: Merge output of two job in a single output file

why just not append to first file?

-----------
Highlighted
Six Stars

Re: Merge output of two job in a single output file

Vinayak,

 

If the schema for both the files/flows are same you can use tUnite component to combine the result set.

 

In case of different schema, you could append the resultset.

 

In the first file remove "Throw an error if the file exist" in advanced setting of tFileOutputDelimited so the the file will be overwritten for every iteration. make sure "append" is unchecked in basic settings.

 

In the second file should be appended by checking "append" in basic settings.

 

Regards,

Ragu

Highlighted
Six Stars

Re: Merge output of two job in a single output file

How will we append that, will you please guide me or suggest me.
Highlighted
Six Stars

Re: Merge output of two job in a single output file

 

In tFileOutputDelimited_4,  go to basic settings then check (tick) the "Append" and make sure the

"File Name" is same for both tFileOutputDelimited_3. below is the screen shot.

 

tFileOutputDelimited_4.png

Highlighted
Six Stars

Re: Merge output of two job in a single output file

Hi arragupathy,

 

As per your suggestion i have appended successfully. Now i got a output as a delimited file:

GHdelimitedFile.png

Now i have to generate a json file of it, like this:

 

GHjsonData.pngIs there any way to achieve it, as this file having no schema define. Thanks in advance. 

Highlighted
Forteen Stars

Re: Merge output of two job in a single output file


@vinayak019 wrote:

Hi arragupathy,

 

As per your suggestion i have appended successfully. Now i got a output as a delimited file:

GHdelimitedFile.png

Now i have to generate a json file of it, like this:

 

GHjsonData.pngIs there any way to achieve it, as this file having no schema define. Thanks in advance. 


it possible to write java code for this

https://dzone.com/articles/how-to-convert-csv-to-json-in-java

https://stackoverflow.com/questions/19766266/directly-convert-csv-file-to-json-file-using-the-jackso...

 

or use something ready to use -

https://csvkit.readthedocs.io/en/1.0.3/ 

https://csvkit.readthedocs.io/en/1.0.3/scripts/csvjson.html?highlight=csvjson

 

-----------
Highlighted
Six Stars

Re: Merge output of two job in a single output file

Hi Vapukov,

 

I was using your suggested approach in https://community.talend.com/t5/Design-and-Development/need-to-convert-delimited-file-to-key-value-p... this.

So my job design like this:csvTOJsonKeyValuePairJob.png

 

 

 

I have replaced '|' delimited with ',', so input file is like that only:inputData.png

 tFileInputDelimited_2 setting:tFileInputDelimited_2Details.png

 

 

tJavaFlex_1 code:tJavaFlex_1.png

 

 

now again, OnComponentOk, tFileInputDelimited_3 setting:tFileInputDelimited_3Details.png

 

 

tjavaFlex_2 setting:tJavaFlex_2.png

 

 

tJavaFlex_3 setting:

tJavaFlex_3.png

 

 

output of this job is:JsonOutput.png

 

 

Why it is creating json for first record only. my expectation is:


GHjsonData.pngWhere I am doing mistake, please let me know.

 

 

 

 

Highlighted
Forteen Stars

Re: Merge output of two job in a single output file

think You are not read till the end of page 

 

find on page text about delimiter :-)

-----------
Highlighted
Six Stars

Re: Merge output of two job in a single output file

Hi,

Will you please explain a little more, not getting it exactly.

 

Highlighted
Forteen Stars

Re: Merge output of two job in a single output file

because You are attaching as a source my example (sure not the best way, but it was created fast on one of the projects)

 

You can see below on the same page - for convert csv (comma delimited) string, I read from a file as \t (TAB) delimited, in this case, all row it a single column

 

If You check other links, You can create more elegant code of course

 

P.S.

You always could use csvkit, I use it as well from talend for different reasons

-----------

View solution in original post

Highlighted
Six Stars

Re: Merge output of two job in a single output file

hi vapukov,

I am able to generate json file. In place of delimited file I have used tFileInputFullRow.Job design is like that:jobDesignForJson.png

Output of this job is:JsonData.png

 

 

 

Now I have to write the tJavaFlex output in a json file, how can I write that? Any guidance will very much favorable for me. Not able to use tFileOutputJson component, because I have to write tjavaFlex data in that generated output file.

Highlighted
Six Stars

Re: Merge output of two job in a single output file

Hi Vapukov,

 

can you share small job design using CSVKIT, simple small demonstration, will help me in writing code elegantly. Thanks in advance.

Highlighted
Six Stars

Re: Merge output of two job in a single output file

Solved using Jackson library, by creating a routine in talend.

View solution in original post

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog