I have multiple file with different structure.
Is there a way you create one dynamic schema and load all files on the fly.
If not, can it be done using Spark? If yes, how do you do it?
With talend subscription solution, here is a dynamic schema feature which allows you to design schema with an unknown column structure (unknown name and number of columns).
Where are your input files from?
Here is a community article:https://community.talend.com/t5/Design-and-Development/How-to-process-changing-data-structure/ta-p/2...
Let us know if it is what you are looking for.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Find out about Talend's 2019 Summer release
Talend continues to revolutionize how businesses leverage speed and manage scale
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend