Five Stars

dynamic job to load multiple tables from multiple input files

Hi,
I have a scenario where i have to load like 200 target tables from 200 input files having one-to-one mapping. each file structure is different.
i also have an extra column in the input files having values as 'I'/'U'/'D' depending on which i will Insert/Update/Delete the records of the target tables.
now the simple solution is to write 200 different jobs but i'm trying to design this using one single dynamic job considering i have no transformation logic and all.
problems i am facing:
1. need to change the schema of tfileinputdelimited dynamically.
2. need to write a metadata file/table for the primary keys of different tables for update/delete.
3. these primary keys will also have to change dynamically.
being a novice in the world of talend i will really appreciate any help.
many thanks in advance.
poulami
2 REPLIES
Moderator

Re: dynamic job to load multiple tables from multiple input files

Hi,
Here is dynamic schema feature which allows you to design schema with an unknown column structure (unknown name and number of columns).
Could you please take a look at this KB article:TalendHelpCenter: How to process changing data structure?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: dynamic job to load multiple tables from multiple input files

Hi sabrina,
This one loads table from one single file only where only the schema type is selected as dynamic. 
But in my scenario  i have 200 source files and 200 respective target tables. 
Also, i don't want to use 'Create table' as my table is already present in the database and my file contains two extra fields (dml_flag and datetime) as i mentioned earlier.
According to my understanding i need to loop through all source files , check dml_flag column for insert/update/delete and perform the operation accordingly for the respective target tables. And this whole process is to be dynamic as all the source and target schema are different.