Five Stars

How to read the data from the failure

Hi,

 

I have a requirement. In my source we have 1000(example) records and performing logic and loading into database.

The job failed after loading 450 records. Now I want to start loading/ reading the source file from 451 to 1000 and It should not read 1-450(which already loaded). Please suggest how can we achieve this using talend.

 

My Job:

 

tfileinputdeleimeted->tmap->toracledb.

 

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Twelve Stars TRF
Twelve Stars

Re: How to read the data from the failure

Hi,

There is a lot of way for this.

If you just want to ignore the n first records from your input file (that's your request), you may have a context variable set to 1 by default ignore the header if any (set to 0 in case of no header), and set the number of records to ignore (450 in your example) + 1 if you have a header.

Then, use this variable to populate the tFileInputDelimited "Header" field.

That's all.


TRF
6 REPLIES
Twelve Stars TRF
Twelve Stars

Re: How to read the data from the failure

Hi,

There is a lot of way for this.

If you just want to ignore the n first records from your input file (that's your request), you may have a context variable set to 1 by default ignore the header if any (set to 0 in case of no header), and set the number of records to ignore (450 in your example) + 1 if you have a header.

Then, use this variable to populate the tFileInputDelimited "Header" field.

That's all.


TRF
Six Stars

Re: How to read the data from the failure

Hi,

There are two ways to achieve this.

 

1) If you have a unique key in your delimited file, then you can apply a join (inner join) between the file and  the database table in which you are inserting the records. That can be done through tMap. In this way it will escape all the records that are already inserted into the table.

 

2) Its not the ideal way, but you can set the header value to 451 (If you have headers) OR 450 (If you don't have headers). In this way it will escape the first 450 records.

 

Hope it will help you in resolving your query !

 

 

Best Regards,

Abhishek

 

Six Stars

Re: How to read the data from the failure

Hi there,

 

From the information you've provided, and assuming this is a one-off exercise, you just need to temporarily change the Header property in your tFileInputDelimited to 450, as suggested by @TRF ad @abhishek.

 

If there are actual header lines in the source file, then you'll also need to add the number of header lines, so e.g. if you have a two line header, the Header value should be 452.

 

Regards,

 

 

Chris

Five Stars

Re: How to read the data from the failure

Thanks all for your reply..giving the value at header resolved the issue.
Twelve Stars

Re: How to read the data from the failure

I notice that you have 3 components with a tMap being in the middle. Therefore it is likely that your tMap is where the error occurs. Here is a tutorial on how you can trap the error row in a tMap (https://www.rilhia.com/quicktips/quick-tip-how-debug-tmap-errors). 

Rilhia Solutions
Twelve Stars TRF
Twelve Stars

Re: How to read the data from the failure

Did you found an answer to your initial question?
If so, mark your case as solved by choosing the appropriate answer.

TRF