merging multiple files with the same schema

One Star

merging multiple files with the same schema

I have multiple files with the same schema. They are already sorted. I want to merge them together into a single file that is also sorted by the same field.
I can use tUnite --> tSortRow but this is highly inefficient, plus I get an out-of-memory error
Is there a "version" of tUnite that reads multiple files and outputs the records in sorted order?
Suggestions?
Thanks,
--eric
Community Manager

Re: merging multiple files with the same schema

Hello
tUnite is the most suitable component for merging records.
You can try the following way:
1)Go to Windows/Preferences/Talend/RunDebug, and modify the vm argument in "Job Run VM arguments" table.
2)Split your job into two subjob, one is merge all the records into a temporary file, another is extract records from temporary file and sort them, output them to target file.
Best regards

shong
----------------------------------------------------------
Talend | Data Agility for Modern Business

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch