[resolved] Filter in the source

[Anonymous]

Hi team,
Now, I got two sources by tMSSqlInput and joined them in the tMap. I encountered a performance issue when i ran the subjob. The issue is which need to get all data than go to join. I think it's very slow. 
I want to do the filter in the tMSSqlInput, but i am not sure whether i can do it or not. Or do you have any idea for the issue?

This is script that i can NOT do the join in it.

Only do the join in the tMap.

Please help me  to review it, thanks.

[Anonymous]

Hi,
The tMap component needs load all data into local firstly and then doing match.
Depending on the filters you are using you could use ELT components to let the DBMS handle the filtering. Have you already checked talend ELT components for sql server? tELTMSsqlInput, tELTMSSqlOutPut and tELTMSSqlMap.
Best regards
Sabrina

[Anonymous]

I think i can NOT use tELTMSSql component because my source data is from  different Database.

[vapukov]

Filter in tMySQLInput - it is normal WHERE in SQL code - so this is You can do easily,
but main idea of ELT - load data to destination and than compare and transform
ok, You are have 50 M in Lookup table, but how many in new(LoadPackage...)?

[Anonymous]

Hi vapukov,
There are about 1000 rows in LoadPackage... which just is one day data.
But in the LoadRateFulfill.... WHERE conditions are from LoadPackage... result, so i have no idea how to do it.

[Anonymous · Accepted solution]

I have fixed the issue, just share my solution.
In the tMap , set Lookup Model to "Reload at each row" and add golobalMap variables.

Filter in tMSSqlInput, use WHERE in SQL code with golbalMap.