I want to achieve a situation where I have different sources(tables and files) all are having the different structure. So, need a way to import these sources dynamically or with a metadata driven approach and able to transform the columns as well by applying any business logic before loading into target.
Source table-1(structure gets changed on daily basis) --> target file 1
Source file-1(structure gets changed on daily basis) --> target file 2
Kindly suggest how we can achieve this in Talend.
NOTE: I'm aware about the Dynamic column functionality in various Talend components but:
1. How can we transform the columns getting in dynamic column
2. And I'm creating a Spark job and it doesn't support this functionality.
Dynamic schema type can be used with components in a DI standard job. But, it is not available with a Big Data Spark job in Talend Studio designer.
Here exists a new feature jira issue:https://jira.talendforge.org/browse/TBD-3302
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Read about OTTO's experiences with Big Data and Personalized Experiences
Pick up some tips and tricks with Context Variables
Take a look at this video about Talend Integration with Databricks