Using open studio 6.4 data integration is it possible to stop processing a file midway through a job if certain conditions are met? I need to re-design a talend job which attempts in ingest excel files but where the structure of the data can change between files.
At the moment my job attempts to recover from these changes by re-reading the file in an attempt to match the ingested file against another schema but this way seems to end with me repeatedly executing components x times where x is the number of rows incorrectly matching the schema.
I am ingesting my data as strings but I'm finding it difficult to understand the best way of handling working with data where the data within a file is consistent when compared to a schema but not necessarily consistent across files
This might help you figure out a way to identify/handle the switching schema
The problem is I can't be sure the column names would stay the same, some reports have more columns than others, a column may disappear for a few months then return back
Is there a way to break out of/away from processing a particular flow and start another?
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.