Table comparison between Impala Vs SQL server

One Star

Table comparison between Impala Vs SQL server

Hello,
I need to compare tables in Impala vs table in SQL server. I am currently using tmap component to compare the tables.As we have  200million records to compare it is taking around 2 days to complete.
My flow is as below :-
tsqlinput and tjdbcinput connected to tmap and results in excel.
Is there any other approach in talend to compare impala tables vs sql server to complete the comparison faster.
Regards,
Raakesh R
Employee

Re: Table comparison between Impala Vs SQL server

i would suggest sqoop to move the sql data to the cluster and then doing the comparison on the cluster.  that should allow you to scale the job and radically reduce your process time.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now