I am trying to use tMap with a large lookup dataset and I run out of memory. I have increased the amount but I need to understand how to architect a solution that does not have a restriction on ram. One method I was thinking about was to put a tloop around a tmap component and chunk the lookup dataset to 5 million rows. I could use the head and limit function of the tfileinput and use tloop as a variable. I just cant figure out where to connect the tloop iterate branch around the tmap component. I am looking for direction on this or if I am way off base on how to approach this issue I am open to new ideas as well.
This is always going to be trade off between speed and memory efficiency. So long as you can put up with the process potentially getting a little slower, you can make your lookups targeted so that you do not have the memory issues. You need to set your lookup model to be "reload at each row" and then set up the globalMap variable to be your lookup key. The simply use that variable to filter your query for the lookup. I demonstrate this (kind of) in this tutorial (
https://www.rilhia.com/quicktips/quick-tip-row-multiplication). Where I am using the globalMap to trigger some functionality in a tJavaFlex, you need to use it modify the WHERE clause of your query.
Hi All,
I am also facing same issue. Our job ran for some time and then out of memory error comes.
I have implement
tMap settings
Store temp data
option as true, tried to execute job it fails instantly. It not even read single record from lookup table.
I am using talend mdm platform 6.1.1 studio.
Request all to help me on same issue.
Thanks,
Vivek
With your talend MDM platform product, could you please report a ticket on talend support portal so that we can give you a remote assistance on your outof memory issue through support cycle with priority?
Best regards
Sabrina
