Table comparison between Impala Vs SQL server

One Star

Table comparison between Impala Vs SQL server

Hello,
I need to compare tables in Impala vs table in SQL server. I am currently using tmap component to compare the tables.As we have  200million records to compare it is taking around 2 days to complete.
My flow is as below :-
tsqlinput and tjdbcinput connected to tmap and results in excel.
Is there any other approach in talend to compare impala tables vs sql server to complete the comparison faster.
Regards,
Raakesh R
Employee

Re: Table comparison between Impala Vs SQL server

i would suggest sqoop to move the sql data to the cluster and then doing the comparison on the cluster.  that should allow you to scale the job and radically reduce your process time.

What’s New for Talend Spring ’19

Join us live for a sneak peek!

Sign up now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch