topic Re: tMap with large lookup in Talend Studio

tMap with large lookup

Anonymous — Wed, 24 Aug 2016 04:56:41 GMT

I am trying to use tMap with a large lookup dataset and I run out of memory. I have increased the amount but I need to understand how to architect a solution that does not have a restriction on ram. One method I was thinking about was to put a tloop around a tmap component and chunk the lookup dataset to 5 million rows. I could use the head and limit function of the tfileinput and use tloop as a variable. I just cant figure out where to connect the tloop iterate branch around the tmap component. I am looking for direction on this or if I am way off base on how to approach this issue I am open to new ideas as well.

Re: tMap with large lookup

Anonymous — Wed, 24 Aug 2016 08:21:39 GMT

Hi,
Have you already checked document about: TalendHelpCenter:Exception outOfMemory?
Best regards
Sabrina

Re: tMap with large lookup

Anonymous — Wed, 24 Aug 2016 08:55:59 GMT

This is always going to be trade off between speed and memory efficiency. So long as you can put up with the process potentially getting a little slower, you can make your lookups targeted so that you do not have the memory issues. You need to set your lookup model to be "reload at each row" and then set up the globalMap variable to be your lookup key. The simply use that variable to filter your query for the lookup. I demonstrate this (kind of) in this tutorial ( https://www.rilhia.com/quicktips/quick-tip-row-multiplication). Where I am using the globalMap to trigger some functionality in a tJavaFlex, you need to use it modify the WHERE clause of your query.

Re: tMap with large lookup

Anonymous — Wed, 24 Aug 2016 10:58:20 GMT

Hi All,
I am also facing same issue. Our job ran for some time and then out of memory error comes.
I have implement tMap settings Store temp data option as true, tried to execute job it fails instantly. It not even read single record from lookup table.
I am using talend mdm platform 6.1.1 studio.
Request all to help me on same issue.
Thanks,
Vivek

Re: tMap with large lookup

Anonymous — Thu, 25 Aug 2016 11:07:09 GMT

Hi,

I am using talend mdm platform 6.1.1 studio.

With your talend MDM platform product, could you please report a ticket on talend support portal so that we can give you a remote assistance on your outof memory issue through support cycle with priority?
Best regards
Sabrina