How to handle frequently changing structure of a data source?

Six Stars

How to handle frequently changing structure of a data source?

Hi,
I want to achieve a situation in which data to be moved from source table(RDBMS or flatfile) to target table(S3).

1. Source Data Set -- Let say on day 1, there are 5 columns in source table so want to move these 5 columns into one target table.

But on day 2, 3 new columns are added to same source table. So, I want all these 8 columns of source table will be picked up dynamically without defining or altering any schema. And, all 8 columns will get defined in the same target table without defining or altering any schema.

 

I want to achieve this situation with a dynamic solution. So that no need to define or alter the schema both at source and target end.

 

Can anyone please suggest.....

Highlighted
Employee

Re: How to handle frequently changing structure of a data source?

Hi,

 

    In Talend, the schema of source and target DB cannot be modified dynamically. 

 

Warm Regards,

 

Nikhil Thampi

Six Stars

Re: How to handle frequently changing structure of a data source?

Hi, 
I'm using the commercial version of Talend for BigData Platform and aware about the Dynamic schema functionality as well which provide me the way to cater the n number of columns dynamically.

But I want to achieve if the columns changes in source side then I'll be able to perform some transformations over it then able to store or transfer the to the target with the dynamic schema only.

 

Anyone worked on such requirement previously or suggest......

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch