topic Re: Tmap Large Lookup in Talend Studio

Tmap Large Lookup

Prasanna3 — Sat, 16 Nov 2024 02:48:15 GMT

Hi Team,

I am doing Lookup with 2 crores data .

What are the points needs to consider to improve performance?

Can someone please suggest me solution

Re: Tmap Large Lookup

manodwhb — Fri, 03 Apr 2020 09:08:39 GMT

Use store on temp and increase no of max rows in tmap settings.

Re: Tmap Large Lookup

Prasanna3 — Fri, 03 Apr 2020 09:13:47 GMT

I have 8gb memory

1) still I need to use store temp
2)I have 3 cores data then how much should I increase no of max rows in tmap settings

Re: Tmap Large Lookup

manodwhb — Fri, 03 Apr 2020 09:19:56 GMT

Check the below links.
https://community.talend.com/t5/Design-and-Development/Performance-improvement-lookup-with-tmap/td-p/142029

https://community.talend.com/t5/Design-and-Development/Improve-performance-when-do-a-lookup/td-p/23630

Re: Tmap Large Lookup

Prasanna3 — Tue, 07 Apr 2020 03:29:26 GMT

Hi,

Main Flow:
PID, Start key
12345,201901
123456,201601

Lookup Flow:
PID,CstartKey
12345,201901
12345,201901

Because of having duplicate PIDs in lookup I am getting duplicate records in output is there anyway to eliminate duplicates to output in tmap.

Re: Tmap Large Lookup

manodwhb — Tue, 07 Apr 2020 03:32:03 GMT

From the look-up data before you connect to tmap, if it is db you can write a query to eliminate duplicates or you can use tuniquerow component to remove duplicates.

Re: Tmap Large Lookup

Prasanna3 — Tue, 07 Apr 2020 03:36:10 GMT

Thanks for quick response.

I have to take main Flow PID and
need to Lookup in that table PiD if it matches that matched Startkey need to check between Lookup cstartkey if it is matched then status 'active' else 'inactive'.

Re: Tmap Large Lookup

manodwhb — Tue, 07 Apr 2020 03:40:22 GMT

You need to write expression below way after doing look-up on PID. Starkey and cstartkey are string data types.

row1.Startkey.equals("row2.cstartkey")?"active":"inactive"

Re: Tmap Large Lookup

Prasanna3 — Tue, 07 Apr 2020 03:58:54 GMT

If I do join I am getting duplicates please find the screenshot
IMG_20200407_082748345.jpg

Re: Tmap Large Lookup

manodwhb — Tue, 07 Apr 2020 04:07:21 GMT

from the main and look-up you have to eliminate duplicates and so the join.

Re: Tmap Large Lookup

Prasanna3 — Tue, 07 Apr 2020 04:09:53 GMT

I want all the records from the main Flow but from the lookup I just needed to find out the main Flow date between lookup. Start and end date keys for the matched PID if matched then need to update status in tmap output

Re: Tmap Large Lookup

manodwhb — Tue, 07 Apr 2020 04:49:43 GMT

Based on your last screen shot you have duplicates from main flow also. When you have duplicates the no of records will increase.

Re: Tmap Large Lookup

Prasanna3 — Wed, 08 Apr 2020 03:59:39 GMT

I have 1 crore data is it good option to store data into thashoutput and tbufferoutput for further processing.

Can you please suggeste maximum how much data we can store in hashcomponents to get better performance?

Re: Tmap Large Lookup

manodwhb — Wed, 08 Apr 2020 04:22:16 GMT

tHashoutput store the data on RAM,so based on your RAM you have see is it working.

Re: Tmap Large Lookup

Prasanna3 — Fri, 10 Apr 2020 07:54:08 GMT

I have two subjobs

In first job I am strong data in thashoutput
After first subjob completed then job got failed and job exited in that case thashoutput data will be cleaned or still exist in the memory

Re: Tmap Large Lookup

manodwhb — Fri, 10 Apr 2020 09:17:28 GMT

If you used with in the job it will be available in memory which you can read using thashinput.

You can not use the way in different sub jobs.

Re: Tmap Large Lookup

Prasanna3 — Fri, 10 Apr 2020 09:32:48 GMT

I am using thashinput in different subjob

Re: Tmap Large Lookup

manodwhb — Fri, 10 Apr 2020 09:33:39 GMT

Which will not able to get the data right?