Best Practice regarding reading CSV file and duplicates

Four Stars

Best Practice regarding reading CSV file and duplicates

Hi

 

I was hoping I could get some suggestions for best practice.  A system generates multiple CSV for applications, each time with a different name based on time.  I understand I can use tFileList to read from multiple CSV.  I need to read these CSV's and insert them into Salesforce (new records, not update).  How can I ensure I am not creating duplicate records in Salesforce by not reading from the same CSV multiple times?


Accepted Solutions
Sixteen Stars TRF
Sixteen Stars

Re: Best Practice regarding reading CSV file and duplicates

Rename or move a file as soon as you have finish with it. This the simplest approach.

TRF

View solution in original post


All Replies
Sixteen Stars TRF
Sixteen Stars

Re: Best Practice regarding reading CSV file and duplicates

Rename or move a file as soon as you have finish with it. This the simplest approach.

TRF

View solution in original post

Four Stars

Re: Best Practice regarding reading CSV file and duplicates

Thanks for the suggestion.  I looked it up, fFileCopy and remove source file should work.

 

https://community.talend.com/t5/Design-and-Development/Move-File-from-one-folder-to-another-folder/t...

Sixteen Stars TRF
Sixteen Stars

Re: Best Practice regarding reading CSV file and duplicates

That's it, the same component gives you the choice.

TRF

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog