No duplicates entry while inserting data

Six Stars

No duplicates entry while inserting data

Hello,

I have CSVs and there are lots of data in CSVs. Basically when I ran job for the specific client I need not to insert the duplicate entry which have same billing number.

 

Untitled.png

This is IgnoreRecord in which I need a condition that same billing number enter not inserted in the table.

The sample data is like this:
Customer,,Calendar day,Billing document,Reference document,Customer PO,    Net Qty,"    Avg
Invoice
Price","    Invoice
Price"
199,Test,2018-01-08,2301404902,66184874,M-1-4-18,-1,$55.00 ,($55.00)
199,Test,2018-02-19,2117199105,532021306,120493,79,$45.00 ,"$3,555.00 "
199,Test,2018-02-19,2117199105,532021306,120493,32,$45.00 ,"$1,440.00 "

 

The bold one is the billing number.
So basically I need only 2 entries in the table when I ran the code. No need to insert same billing number entry.

Community Manager

Re: No duplicates entry while inserting data

Use a tAggregateRow to achieve this. Connect it to your input component and group by your billing number column. Output ALL of the other columns in your "Operations" table and set the Function for each column to First or Last. The Function allows you to specify whether you want the first record in the group's values or the last record in the group's values to be used. There are other functions, but I think you will probably only need to look at those.

 

 

Fifteen Stars TRF
Fifteen Stars

Re: No duplicates entry while inserting data

Or just a simple tUniqRow.

TRF
Community Manager

Re: No duplicates entry while inserting data

The tUniqRow would work, but it doesn't give you the sort of control over the values to keep that the tAggregateRow does. I suggested it with the fact in mind that the "duplicate rows" are not truly duplicate and therefore I'd expect preferred values from the two or more rows to be required.

Fifteen Stars TRF
Fifteen Stars

Re: No duplicates entry while inserting data

One more great explaination from @rhall_2_0

TRF

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog