Remove duplicates from a MySQL table

Seven Stars

Remove duplicates from a MySQL table

Coming from SAS background, I am finding this basic task of deduping a dataset quite a chore with Talend. I am certain that it's to do with the lack of my experience with the tool.

 

I need to remove duplicates from a target dataset, after inserting the data. I can't seem to run multiple sql statements using tSortRow (MySQL) and I get the error - "You have an error in your SQL syntax". The set of queries work fine on the MySQL Workbench.

 

With SAS Data Management Studio, I could add a SAS code node and could run this sort procedure with noduplicates modifier. I was trying to do something similar by running the set of SQL queries, through tMySQLRow component.

 

How do you guys do that? All I need is the talend way to deduplicate the data in the datasource.

Seven Stars

Re: Remove duplicates from a MySQL table

Hello sushantV,

 

to run multiple SQL statements in one tMysqlRow, you have to set an additional jdbc parameter (allowMultiQueries) in advanced settings:

 

tmysqlrow_jdbc_parameters.jpgjdbc parameters

 

Best regards,

 

Thomas

 

Employee

Re: Remove duplicates from a MySQL table

Hi,

 

    Could you please try tUniqrow component for your use case? 

 

     Please refer the help document of this component?

 

https://help.talend.com/reader/iJZ9g~EoO2NwssXWselCkQ/KDI5XVvKJYiPPiV0cKGtOw

 

Warm Regards,

 

Nikhil Thampi

Seven Stars

Re: Remove duplicates from a MySQL table

Hi sushantV,

 


@sushantV wrote:

I was trying to do something similar by running the set of SQL queries, through tMySQLRow component.

 


The tMysqlRow component is not a component that provides output. It can execute a query (or multiple queries) for each input row, but it does not give you data like the tMysqlInput component.

 

Best regards,

 

Thomas

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

APIs for Dummies

View this on-demand webinar about APIs....

Watch Now