One Star

Table comparison between Impala Vs SQL server

Hello,
I need to compare tables in Impala vs table in SQL server. I am currently using tmap component to compare the tables.As we have  200million records to compare it is taking around 2 days to complete.
My flow is as below :-
tsqlinput and tjdbcinput connected to tmap and results in excel.
Is there any other approach in talend to compare impala tables vs sql server to complete the comparison faster.
Regards,
Raakesh R
1 REPLY
Employee

Re: Table comparison between Impala Vs SQL server

i would suggest sqoop to move the sql data to the cluster and then doing the comparison on the cluster.  that should allow you to scale the job and radically reduce your process time.