Splitting data to separate files, dynamically

Highlighted
One Star

Splitting data to separate files, dynamically

Hi!
I have a data in table that need to be exported into different files, based on value in "category" field.
The number of distinct values may be dynamic (e.g. cannot use tMap component and hard-code the values).
File name will include the category string.
Columns:
ID | VALUE1 | VALUE2 | CATEGORY
1 abc abc A
2 abc xyz B
3 abc abc A
According to example above, 2 files should be created , e.d. /data/file_A.csv, /data/file_B.csv
Any suggestions?
Thanks!
Highlighted
Moderator

Re: Splitting data to separate files, dynamically

Hi gadib,
Is this related forum working well for your use case?
Forum 28035Smiley Frustratedplit a input file into multiple output files
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Four Stars

Re: Splitting data to separate files, dynamically

Hi Gabib,
Use tfileinput-->tMap--(Get two output by applying a filter condition for your records)
This would help
Thanks
Vaibhav
Highlighted
One Star

Re: Splitting data to separate files, dynamically

Thank you.
The link by xdshi is useful, however it implies that the large table with data will be selected several times, for each distinct value in "category" field. Which is very not effective.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog