how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Eight Stars

how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi All

I have one requirement I have different source like csv or xml or excel it may not be know to us always the source will keep on changing and schema also varies. in this case how can i handle the file to insert the records in database which is having fixed schema . by using Talend open studio for data integration.

waiting for your reply

Regards 

rekha  

Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi ,

 

Can you confirm whether the schema is fixed for specific file type(e.g.-xml file contains always same schema) ?

 

Thanks,

Premchand

Eight Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

yes schema also varies for each file each time

Eight Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

can any one give reply for this requirement please.is it possible

Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi ,

 

As of now this feature is not working in talend,we also tried a lot to achieve this but no luck 

 

Thanks,

Premchand

Four Stars PVK
Four Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

I hope this post gives you an idea on dealing with dynamic schemas.

http://bekwam.blogspot.fr/2011/06/dynamic-schemas-in-talend-open-studio.html 

Six Stars PK
Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

 

We have done similar requirement like this but in our case we were getting only one type of data file in csv and dynamic schema file.

Eight Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

so ,can you share how you have handled in your case if it is only csv file


@PK wrote:

Hi,

 

We have done similar requirement like this but in our case we were getting only one type of data file in csv and dynamic schema file.


 

Eight Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

it is showing blog does not exist

Six Stars PK
Six Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

 

Idea is to :

1. first read the files using tfilelist component.

2. Read the header and create temp table in db using header with varchar type for all columns.

3.load the data into temp table.

4. read schema file using tfilelist.

5.Create actual table .

6. insert into acutal table from temp table.

 

Regards

PK

 

 

Seven Stars skh
Seven Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Hi,

1) Use tFileList Component --> tIterateToFlow --> tSetGlobal (Here u just set the Parameter's of tFileList like FileName,FileCurrentPath & ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION")) 

2) In 2nd Sub-Job Create 3-different Sub-Job Flow's which can Handle the 3-different FileTypes.

3) Connect the tSetGlobal Component to Each one of the 3-Flows Separately with Trigger --> RunIf Condition's 

tSetGlobal_1 --> If(Order 1) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "xlsx"

tSetGlobal_1 --> If(Order 2) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "csv"

tSetGlobal_1 --> If(Order 3) [Set Condition in if as ((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION"))  == "xml"

4) So whenever tFileList Found the FileExtension Type (csv,xlsx or xml) then it will execute that particular sub-job.

 

I hope this will help you.

Thanks,

skh

 

Eight Stars

Re: how to handle dynamic schema files which come on daily basis to load into fixed schema databse

Thanks for reply i will do and let you know the status

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog