[resolved] Filter in the source

Six Stars Icy
Six Stars

[resolved] Filter in the source

Hi team,
Now, I got two sources by tMSSqlInput and joined them in the tMap. I encountered a performance issue when i ran the subjob. The issue is which need to get all data than go to join. I think it's very slow. 
I want to do the filter in the tMSSqlInput, but i am not sure whether i can do it or not. Or do you have any idea for the issue?

This is script that i can NOT do the join in it.

Only do the join in the tMap.

Please help me  to review it, thanks.

Accepted Solutions
Six Stars Icy
Six Stars

Re: [resolved] Filter in the source

I have fixed the issue, just share my solution.
In the tMap , set Lookup Model to "Reload at each row" and add golobalMap variables.

Filter in tMSSqlInput, use WHERE in SQL code with golbalMap.

All Replies
Moderator

Re: [resolved] Filter in the source

Hi,
The tMap component needs load all data into local firstly and then doing match.
Depending on the filters you are using you could use ELT components to let the DBMS handle the filtering. Have you already checked talend ELT components for sql server? tELTMSsqlInput, tELTMSSqlOutPut and tELTMSSqlMap.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars Icy
Six Stars

Re: [resolved] Filter in the source

I think i can NOT use tELTMSSql component because my source data is from  different Database.
Twelve Stars

Re: [resolved] Filter in the source

Filter in tMySQLInput - it is normal WHERE in SQL code - so this is You can do easily,
but main idea of ELT - load data to destination and than compare and transform
ok, You are have 50 M in Lookup table, but how many in new(LoadPackage...)?
-----------
Six Stars Icy
Six Stars

Re: [resolved] Filter in the source

Hi vapukov,
There are about 1000 rows in LoadPackage... which just is one day data.
But in the LoadRateFulfill.... WHERE conditions are from LoadPackage... result, so i have no idea how to do it.
Six Stars Icy
Six Stars

Re: [resolved] Filter in the source

I have fixed the issue, just share my solution.
In the tMap , set Lookup Model to "Reload at each row" and add golobalMap variables.

Filter in tMSSqlInput, use WHERE in SQL code with golbalMap.