I need a help to find loop handling
I have 5 files with 100 records each.
I want to take 1st file and load 10 records as loop and load into db. Once first file is over then 2nd file and again load 10 records as loop and load into db. It should iterate for all 5 files.
Can anyone help me?
Thanks in Advance
Thanks for your response
If I am setting "Limit" as 10 then its taking only 1st 10 record of each file.
But I want this to read full file means 100 records so 10 times it should iterate then it should go for 2nd file
Sorry, I misread your request.
The easiest way is to split the input files into 10-line files.
The process should look like this:
tFileList-->tFileInputDelimited-->tFileOutputDelimited | onSubjobOK tFileList-->tFileInputDelimited-->t<DBName>Output
In 1rst subjob, you iterate over the input files and push the result into small output files.
In the tFileOutputDelimited, tick the option "Split output in several files" and enter 10 into the field "Rows in each output file".
In the 2nd subjob you will iterate over the list of "small" input files, read the content and push it into your database.
Records - 25 Million
approximately 10 files
25 Million is one time but 3 Million weekly
I need to process in batch
Can you give some suggestion?
I strongly suggest you to try the suggested approach.
3 million records per week, not a so huge volume, and Talend is very fast with this kind of files, it could be a good solution despite of the opposition regarding the i/o operations.
Else, you will have to play with tFlowToIterate / tLoop / tIterateToFlow to build the right algorithm.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema