track changes

Highlighted
Five Stars

track changes

I have developed a job thats supposed to track changes by comparing two sources and create an output insert. My tMap has a variable expression defined as below. My records are in millions of rows and performance it not very good. Any ideas please. - thank you in advance.

 

source1.customerid.equals(source2.customerid) && source1.hashing.equals(source2.hashing)? "no change":"changed"

 

Tags (1)

Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: track changes

It seems you don't have a join between input rows (not sure because you have reduce everything). You should (must) have a left outer join then, depending on result you'll decide which output flow to address.

TRF

All Replies
Fifteen Stars TRF
Fifteen Stars

Re: track changes

Can you share the tMap?

What kind of join do you have between the main and the lookup row?


TRF
Five Stars

Re: track changes

Here is my tMap

Five Stars

Re: track changes

mgs.png

Five Stars

Re: track changes

I have posted it below.
Fifteen Stars TRF
Fifteen Stars

Re: track changes

It seems you don't have a join between input rows (not sure because you have reduce everything). You should (must) have a left outer join then, depending on result you'll decide which output flow to address.

TRF

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now