One Star

Multiple tFileList

Hi all,
I have multiple tFileList in one job and the component will iterate all Excel files in one folder.
The problem is the output I get is not updated after execute each files. I wonder what is wrong with my job?
Thank you in advance!
8 REPLIES
Community Manager

Re: Multiple tFileList

hi
What is your problem? You don't describe it clearly in your post, what are you expected result?
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Multiple tFileList

Hi Shong,
I have three tFileList components for my job. At each tFileList, the path is directed to a folder which contained 2 excels file.
I want the job to iterate reading and process both files and display output.
I expect the output will display distinct value from both Excel files. However, the output is not 100% correct. Some are correct some are still contained data from the first Excel.
My current job output:
1st Excel
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
2nd Excel
--------------
Table10|2009|22
Table10|2009|21
Table10|2009|20
Expected output:
1st Excel
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
2nd Excel
--------------
Table11|2006|22
Table11|2006|21
Table11|2006|20
Could you help me on this?
Appreciate your feedback. Thanks.
Community Manager

Re: Multiple tFileList

Why don't you use two tFileList to iterate the same folder? From your image, I don't see you define the file mask if you want each tFilelist to iterate a group of specified files.
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Multiple tFileList

Hi Shong,
From my understanding of Rozie's requirement, she wants to extract the files to be like this:
1st Excel
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
2nd Excel
--------------
Table10|2009|22
Table10|2009|21
Table10|2009|20

Expected Result
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
Table11|2006|22
Table11|2006|21
Table11|2006|20
But, after execute the job, she got the output as:
Current Result
--------------
Table11|2006|11
Table11|2006|12
Table11|2006|13
Table11|2006|22
Table11|2006|21
Table11|2006|20
You can see the table id and year was repeated. As your suggestion to use two different tFileList, i dont think so because we have so many files for respective year. This is an example for two year files. If we have 100 years, then are we need 100 tFileList? Because the files with the same format and structure and the only different are the value of each years. Hope you understand the requirement.
Regards,
Kal
Community Manager

Re: Multiple tFileList

Hi Kal
I don't suggest Rozie to use different tFileList component, I also felt surprise why she use two tFileList with no file mask in her case, and she did't explain why the table id and year was changed? any transformation?
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Multiple tFileList

Hi Shong, I'm sorry there's misunderstanding, here is the correction:
1st Excel
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
2nd Excel
--------------
Table11|2006|22
Table11|2006|21
Table11|2006|20

Expected Result
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
Table11|2006|22
Table11|2006|21
Table11|2006|20

Current Result
--------------
Table11|2006|11
Table11|2006|12
Table11|2006|13
Table11|2006|22
Table11|2006|21
Table11|2006|20

Rgds,
Kal
One Star

Re: Multiple tFileList

Hi Shong,
My mistake I didn't put the file mask.
FYI, the table_id and year represent the id of Excel file itself. Like I mentioned before, there are two Excel files in a folder.
Source:
1st Excel
--------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
2nd Excel
--------------
Table11|2006|22
Table11|2006|21
Table11|2006|20
Based on my job image, you can see that I use three tFileList components since I had to put down three times tInputExcel files to extract data accordingly. So, there are three folders pointed at respective tFileList. Each folder contains two Excel files.
What I want is the job to produce result as below (Credits to ae_kel29.):
Expected output
-----------------
Table10|2009|11
Table10|2009|12
Table10|2009|13
Table11|2006|22
Table11|2006|21
Table11|2006|20
But at the moment, my result return this:
Current Result
--------------
Table11|2006|11
Table11|2006|12
Table11|2006|13
Table11|2006|22
Table11|2006|21
Table11|2006|20
Perhaps you can enlighten me to improve my job design or not.
Thanks!
Community Manager

Re: Multiple tFileList

Hi
Based on my job image, you can see that I use three tFileList components since I had to put down three times tInputExcel files to extract data accordingly. So, there are three folders pointed at respective tFileList. Each folder contains two Excel files.

Instead of using three tFileList to read three different folders, I would like to use a tForEach with three folder path to iterate each value if all the files have the same schema, the job looks like:
tForEach--iterate--tFlieList--iterate--tFileInputExcel--main--tLogRow
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business