Six Stars

Loop in folders for files and records

Hi  

 

I need a help to find loop handling 

 

Scenario :

I have 5 files with 100 records each.

I want to take 1st file and load 10 records as loop and load into db. Once first file is over then 2nd file and again load 10 records as loop and load into db. It should iterate for all 5 files.

 

Can anyone help me?

Thanks in Advance 

 

11 REPLIES
Forteen Stars TRF
Forteen Stars

Re: Loop in folders for files and records

Hi,
Use tFileList to iterate over the file list, then tFileInputDelimited (or other depending on the files type) to get the records.
In this component, use the variable tFileList_1_CURRENT_FILEPATH as the filename and set the field "Limit" to the number of records you want to read.

TRF
Six Stars

Re: Loop in folders for files and records

Thanks for your response 

 

If I am setting "Limit" as 10  then its taking only 1st 10 record of each file.

 

But I want this to read full file means 100 records so 10 times it should iterate then it should go for 2nd file

Forteen Stars TRF
Forteen Stars

Re: Loop in folders for files and records

Sorry, I misread your request.

The easiest way is to split the input files into 10-line files.

The process should look like this:

tFileList-->tFileInputDelimited-->tFileOutputDelimited
| onSubjobOK
tFileList-->tFileInputDelimited-->t<DBName>Output

In 1rst subjob, you iterate over the input files and push the result into small output files.

In the tFileOutputDelimited, tick the option "Split output in several files" and enter 10 into the field "Rows in each output file".

In the 2nd subjob you will iterate over the list of "small" input files, read the content and push it into your database.

 


TRF
Six Stars

Re: Loop in folders for files and records

Thanks TRF

 

Splitting file is not allowed here. we want to use same file.

Forteen Stars TRF
Forteen Stars

Re: Loop in folders for files and records

Could you explain why?


TRF
Six Stars

Re: Loop in folders for files and records

We have given suggestion of splitting file but people saying it will cause i/o operation so need to avoid it.

 

So only 

Forteen Stars TRF
Forteen Stars

Re: Loop in folders for files and records

How many files in real life?

How many records?

What's the job frequency?


TRF
Six Stars

Re: Loop in folders for files and records

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

Six Stars

Re: Loop in folders for files and records

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

 

I need to process in batch 

 

Can you give some suggestion?

Forteen Stars TRF
Forteen Stars

Re: Loop in folders for files and records

I strongly suggest you to try the suggested approach.

3 million records per week, not a so huge volume, and Talend is very fast with this kind of files, it could be a good solution despite of the opposition regarding the i/o operations.

Else, you will have to play with tFlowToIterate / tLoop / tIterateToFlow to build the right algorithm.


TRF
Six Stars

Re: Loop in folders for files and records

Thanks TRF

 

tFlowToIterate / tLoop / tIterateToFlow

 

how to use mentioned component and where ?