How to handle frequently changing structure of a data source?

Highlighted
Six Stars

How to handle frequently changing structure of a data source?

Hi,
I want to achieve a situation in which data to be moved from source table(RDBMS or flatfile) to target table(S3).

1. Source Data Set -- Let say on day 1, there are 5 columns in source table so want to move these 5 columns into one target table.

But on day 2, 3 new columns are added to same source table. So, I want all these 8 columns of source table will be picked up dynamically without defining or altering any schema. And, all 8 columns will get defined in the same target table without defining or altering any schema.

 

I want to achieve this situation with a dynamic solution. So that no need to define or alter the schema both at source and target end.

 

Can anyone please suggest.....

Highlighted
Employee

Re: How to handle frequently changing structure of a data source?

Hi,

 

    In Talend, the schema of source and target DB cannot be modified dynamically. 

 

Warm Regards,

 

Nikhil Thampi

Six Stars

Re: How to handle frequently changing structure of a data source?

Hi, 
I'm using the commercial version of Talend for BigData Platform and aware about the Dynamic schema functionality as well which provide me the way to cater the n number of columns dynamically.

But I want to achieve if the columns changes in source side then I'll be able to perform some transformations over it then able to store or transfer the to the target with the dynamic schema only.

 

Anyone worked on such requirement previously or suggest......

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog