Six Stars

How to stop processing file or switch mid job

Using open studio 6.4 data integration is it possible to stop processing a file midway through a job if certain conditions are met? I need to re-design a talend job which attempts in ingest excel files but where the structure of the data can change between files.

 

At the moment my job attempts to recover from these changes by re-reading the file in an attempt to match the ingested file against another schema but this way seems to end with me repeatedly executing components x times where x is the number of rows incorrectly matching the schema.

 

I am ingesting my data as strings but I'm finding it difficult to understand the best way of handling working with data where the data within a file is consistent when compared to a schema but not necessarily consistent across files

Tags (2)
3 REPLIES
Eleven Stars

Re: How to stop processing file or switch mid job

This might help you figure out a way to identify/handle the switching schema

https://www.rilhia.com/tutorials/dynamic-column-order

Rilhia Solutions
Six Stars

Re: How to stop processing file or switch mid job

The problem is I can't be sure the column names would stay the same, some reports have more columns than others, a column may disappear for a few months then return back 

 

Is there a way to break out of/away from processing a particular flow and start another?

Six Stars

Re: How to stop processing file or switch mid job

Hi,

In tjavarow component, you can check the condition and if condition occurs, you can exit there only using System.exit statement. your job will fail.