Six Stars

how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi All

I have one requirement I have different source like csv or xml or excel it may not be know to us always the source will keep on changing and schema also varies. in this case how can i handle the file to insert the records in database which is having fixed schema . by using Talend open studio for data integration.

waiting for your reply

Regards 

rekha  

11 REPLIES
Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi ,

 

Can you confirm whether the schema is fixed for specific file type(e.g.-xml file contains always same schema) ?

 

Thanks,

Premchand

Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

yes schema also varies for each file each time

Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

can any one give reply for this requirement please.is it possible

Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi ,

 

As of now this feature is not working in talend,we also tried a lot to achieve this but no luck 

 

Thanks,

Premchand

Four Stars PVK
Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

I hope this post gives you an idea on dealing with dynamic schemas.

http://bekwam.blogspot.fr/2011/06/dynamic-schemas-in-talend-open-studio.html 

Five Stars PK
Five Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

 

We have done similar requirement like this but in our case we were getting only one type of data file in csv and dynamic schema file.

Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

so ,can you share how you have handled in your case if it is only csv file


PK wrote:

Hi,

 

We have done similar requirement like this but in our case we were getting only one type of data file in csv and dynamic schema file.


 

Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

it is showing blog does not exist

Five Stars PK
Five Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

 

Idea is to :

1. first read the files using tfilelist component.

2. Read the header and create temp table in db using header with varchar type for all columns.

3.load the data into temp table.

4. read schema file using tfilelist.

5.Create actual table .

6. insert into acutal table from temp table.

 

Regards

PK

 

 

Five Stars skh
Five Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

1) Use tFileList Component --> tIterateToFlow --> tSetGlobal (Here u just set the Parameter's of tFileList like FileName,FileCurrentPath & ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION")) 

2) In 2nd Sub-Job Create 3-different Sub-Job Flow's which can Handle the 3-different FileTypes.

3) Connect the tSetGlobal Component to Each one of the 3-Flows Separately with Trigger --> RunIf Condition's 

tSetGlobal_1 --> If(Order 1) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "xlsx"

tSetGlobal_1 --> If(Order 2) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "csv"

tSetGlobal_1 --> If(Order 3) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "xml"

4) So whenever tFileList Found the FileExtension Type (csv,xlsx or xml) then it will execute that particular sub-job.

 

I hope this will help you.

Thanks,

skh

 

Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Thanks for reply i will do and let you know the status