raised new topic

Five Stars

raised new topic

I have used tFilelist to iterate over csv files and process them one by one to database table.

Here I have observed if there is error in any one file processing then the remaining files does not get processed and job gets stuck.

I want the tFilelist iterator to continue with next file after there is error in one csv file.

Tags (1)
Employee

tFileList to continue without error

Hi,

 

    Could you please do a schema compliance check to remove all the bad records before loading to DB. In this way, you can avoid the failure due to data errors.

image.png

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Five Stars

Re: tFileList to continue without error

Appriciate your answer but what if there is constraint violation while insert?? data is fine and validated.
Employee

Re: tFileList to continue without error

Hi,

 

    Since you are truncating the table, you will not have any issues due to previous data. Now in the current data set, if you are having duplicate records, you can take Unique records from them before doing insert itself by using tUniqrow component.

 

    You will have to select the keys from the columns in this component and it will segregate the uniques and duplicates.

 

    Now, even if you are having existing data in your table, you can select insert or update option in "Action on Data" of tDBOutput component and it also will resolve your issue. 

image.png

 

 

Hope I have answered your queries for your topic. Before going, please mark the topic as resolved so that other Talend members also will be benefited from the topic.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Five Stars

Re: tFileList to continue without error

Hi Nikhil,
I don't have to update record with new one. so this solution is not applicable for me anyways Thank You.
Employee

Re: tFileList to continue without error

Hi Roshan,

 

     This requirement was not clear in your previous posts. If you do not want to update any existing record, its quite simple.

 

      Why don't you do a tMap join where lookup is your target table? If you are doing Inner join and have selected inner join reject as True in output section, then only new records will be selected to the output flow.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download