I use the TSqoopimport component for importing oracle tables to HDFS.
This component generate 4 files (part-m-xxxx) if i configure it with 4 mappers.
and after, how can i merge thos 4 files into one file ?
I use TOS for Big Data 6.1.1
Solved! Go to Solution.
The source data comes from 2 different sources, but has the same schema?
You can use a tFileList to iterate on a tFileInput* row linked to a tFileOutputDelimited in append mode.
Let us know if it works.
If the data isn't huge, you can try configuring Sqoop to just use one mapper: that way, it will generate one file.
If you don't want to go that route, Sabrina is mostly correct, except that you'll need tHDFSFileList to iterate over the files. Instead of merging them, this will iterate over them, so you can do whatever ETL work you need to do.
Watch the recorded webinar!
Create systems and workflow to manage clean data ingestion and data transformation.
Introduction to Talend Open Studio for Data Integration.
Test drive Talend's enterprise products.