Hi all, I am doing a mini project which requires me to load 1000+ files from source to target. Each file has a different structure. It is also requires to perform data quality checks on the data before it is loaded into the database. Files are Comma delimited. My main idea is to not to create 1000+ jobs as it will be not easy to maintain. I am looking for re-usable code that can be used for this purpose. For data quality purpose we are good as we have come up with a framework for the same and it will be done at database level. I am more concerned about loading database tables. I have already explored following options: 1. Use Dynamic schema using tSetDynamicSchema component but it seems to be working only for the postional files not delimited files. 2. Referred following links but it didn't really help: http://bekwam.blogspot.in/2011/06/dynamic-schemas-in-talend-open-studio.html https://www.talend.com/blog/2016/03/11/how-to-operationalize-meta-data-in-talend-with-dynamic-schema... Can someone please help with it?
Hi, Usually, we use tfilelist to iterate these files and load them in one target. The work flow looks like: tfilelist(set a filemask )-->iterate>tfileinputdelimited(dynamic schema)-->t<DB>output. The dynamic schema feature will be available in talend subscription solution which allows you to design schema with an unknown column structure (unknown name and number of columns).
Best regards Sabrina
-- Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.