How to check if a record Exists in DB

How to check if a record Exists in DB

I have a folder of excel files
Talend package reads the folder and for each excel file in the folder it unpivots the content of the excel file and inserts in a MSSQL database.
Now the same row/record could exist in multiple files and thats why we keep getting duplicates in the database.
if someone could help me understand, if there is a way to check if the record exists in MSSQL DB, before inserting then i can skip duplicates and just get unique records. Please see the attached screenshot.
Thanks

Re: How to check if a record Exists in DB

Can someone from the expert talend team point me in right direction please. I would highly appreciate it.
One Star

Re: How to check if a record Exists in DB

Can you add a unique key to your database table? If you then set the MSSQL component to be "Insert if not exist", this would stop duplicates being inserted.

Re: How to check if a record Exists in DB

Thank You. Very Helpful. What should i do if i want to update the existing record instead of skipping it? Is there an option in the tMSSQL component for this as well?
One Star

Re: How to check if a record Exists in DB

OK - set the action on data to be "Update or insert". This updates existing entries or create it if non existing.
One Star

Re: How to check if a record Exists in DB

OK - set the action on data to be "Update or insert". This updates existing entries or create it if non existing.

Hi,
For the Update or insert otpion in mysql output it requires a primary key. So how to get the primary keys from db and assign it to the rows in excel sheet. Kindly help.

Thanks,
Senthil
Moderator

Re: How to check if a record Exists in DB

Hi Senthil,
I have replied your related forum Forum 30547, feel free post your issue on forum.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to check if a record Exists in DB

hi

"insert if not exist" option is not available in tverticaoutput component(TOS) how to achieve this case in vertica database.
Moderator

Re: How to check if a record Exists in DB

Hi,
So far, there is no "insert if not exist" option in tverticaoutput.
As alternative, you can set any primary key you wanted in schema setting by manual(checkout the "Key" option) .
and take the "Update or insert" data action in tverticaoutput.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to check if a record Exists in DB

Hi,
I want to convert excel sheet to mysql table.MySQl table has id which is auto increment.
I have list of user in excel sheet which might contain duplicate.I have Insert ignore option which give me correct record but when add new record in existing excel file . auto increment value is almost double. for example 
I have list of user 20 distinct values first time in excel file but i have added one more and try to run same excel file . the auto increment value for new record will be 41 but i want 21. is it posible?
please let me know..its very urgent.
Thank you in advance.
One Star

Re: How to check if a record Exists in DB

You can dump the data from all the excel files in 1 table. Post that you can only move unique or latest records in your actual table.