Tmap Large Lookup

[Prasanna3]

Hi Team,

I am doing Lookup with 2 crores data .

What are the points needs to consider to improve performance?

Can someone please suggest me solution

[manodwhb]

Use store on temp and increase no of max rows in tmap settings.

[Prasanna3]

I have 8gb memory

1) still I need to use store temp
2)I have 3 cores data then how much should I increase no of max rows in tmap settings

[manodwhb]

Check the below links.
https://community.talend.com/t5/Design-and-Development/Performance-improvement-lookup-with-tmap/td-p...

https://community.talend.com/t5/Design-and-Development/Improve-performance-when-do-a-lookup/td-p/236...

[Prasanna3]

Hi,

Main Flow:
PID, Start key
12345,201901
123456,201601

Lookup Flow:
PID,CstartKey
12345,201901
12345,201901

Because of having duplicate PIDs in lookup I am getting duplicate records in output is there anyway to eliminate duplicates to output in tmap.

[manodwhb]

From the look-up data before you connect to tmap, if it is db you can write a query to eliminate duplicates or you can use tuniquerow component to remove duplicates.

[Prasanna3]

Thanks for quick response.

I have to take main Flow PID and
need to Lookup in that table PiD if it matches that matched Startkey need to check between Lookup cstartkey if it is matched then status 'active' else 'inactive'.

[manodwhb]

You need to write expression below way after doing look-up on PID. Starkey and cstartkey are string data types.

row1.Startkey.equals("row2.cstartkey")?"active":"inactive"

[Prasanna3]

If I do join I am getting duplicates please find the screenshot
IMG_20200407_082748345.jpg

[manodwhb]

from the main and look-up you have to eliminate duplicates and so the join.