I want to achieve a situation where I have different sources(tables and files) all are having the different structure. So, need a way to import these sources dynamically or with a metadata driven approach and able to transform the columns as well by applying any business logic before loading into target.
Source table-1(structure gets changed on daily basis) --> target file 1
Source file-1(structure gets changed on daily basis) --> target file 2
Kindly suggest how we can achieve this in Talend.
NOTE: I'm aware about the Dynamic column functionality in various Talend components but:
1. How can we transform the columns getting in dynamic column
2. And I'm creating a Spark job and it doesn't support this functionality.
Dynamic schema type can be used with components in a DI standard job. But, it is not available with a Big Data Spark job in Talend Studio designer.
Here exists a new feature jira issue:https://jira.talendforge.org/browse/TBD-3302
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Learn how to make your data more available, reduce costs and cut your build time