How to skip files which contain error

One Star

How to skip files which contain error

Hi everyone,
I am trying to read multiple XML files from an input directory and then using tMap transform them in desired output format.
While doing so, some of the input files in the directory are having some error (related to XML tags) which is leading my job to fail in between on encountering those non-compliant files. Since the number of files is huge (in the directory), I am not able to check for the files which are non-compliant so that I can remove it. Also, I prefer the job to do this activity so that I can have a log with me.
Can you suggest me a module(s) and arrangement so that my job can skip non-compliant files whenever it encounters one and logs the same?
I am using tFileList to iterate on the directory, then I have tFileInputXMl followed by tMap and tAdvancedFileOutputXML.
Thanks,
Ankur
Community Manager

Re: How to skip files which contain error

Hello
Move the process 'parsing xml file and merging' into a child job, like
on father job:
tFileList--iterate-->tRunJob(run the child job)
on child job:
tFileInputXML--main--->tMap--main-->tAdvancedFileOutputXML
1)Pass the current file path to child job, set it as file path on tFileInputXML.
2)On tRunJob, uncheck the 'die on child error' option.
Best regards
        shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Five Stars

Re: How to skip files which contain error

Hi Shong
May I know what's row component is in child Job ?
Because I'm also doing kind of same task and need to understand this job !!
Community Manager

Re: How to skip files which contain error

Hi ashish123
It is main link here, not a component. I will update my post.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Five Stars

Re: How to skip files which contain error

Hi Shong
I'm working on one talend job and I want whole file to be rejected containing any single error. 
Right now I have a job who checks files record wise and reject only bad one and insert good records from file. But I dont' want to insert anything from file having any single error but reject the whole file. 

 

I also want to store reason why that file got rejected in mysql log table  through tlogcatcher or any other component.
But in this job am storing the rejected records in some table with tschema error message in mysql table.
Please help me for the same.
Best !!
Community Manager

Re: How to skip files which contain error

Hi 
To achieve the request, you have to redesign the job as I shown in my previous post, and you need to implement transaction management with tMysqlConnection+tMysqlcommint and tMysqlRollback.
on father job:
tFileList--iterate-->tRunJob(run the child job)
on child job:

tMysqlConnection
    |
onsubjobok
    |
tFileInputXML--main--->....
   -onsubjobok--tMysqlCommit
   -onsubjoberror--tMysqlRollback
1)Pass the current file path to child job, set it as file path on tFileInputXML.
2)On tRunJob, uncheck the 'die on child error' option.

3)In the child job, use an existing DB connection on tMysqlOutput component.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Five Stars

Re: How to skip files which contain error

Hi Shong
My  Jobs is like this
Parent Job:  tFileList------iterate----->tRunJob
Child Job:  tFileInputDelimited----main---->tMap----main----->tOracleOutput
                - OnSubJobOK  ---->tOracleCommit
                - OnSubJobError---> tOracleRollBack
First Question is that, In job which you had suggested me in your previous post why did you use tOracleConnection ? (am using oracle that's I'm using oracle instead of mysql)
Second is that, Why am getting ora - 00911 invalid character error ? I guess I have settled everything ok at my end but if you think still am missing something please let me know for this error.
Best !! 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog