Using open studio 6.4 data integration is it possible to stop processing a file midway through a job if certain conditions are met? I need to re-design a talend job which attempts in ingest excel files but where the structure of the data can change between files.
At the moment my job attempts to recover from these changes by re-reading the file in an attempt to match the ingested file against another schema but this way seems to end with me repeatedly executing components x times where x is the number of rows incorrectly matching the schema.
I am ingesting my data as strings but I'm finding it difficult to understand the best way of handling working with data where the data within a file is consistent when compared to a schema but not necessarily consistent across files
This might help you figure out a way to identify/handle the switching schema
The problem is I can't be sure the column names would stay the same, some reports have more columns than others, a column may disappear for a few months then return back
Is there a way to break out of/away from processing a particular flow and start another?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema