topic Re: TMap:Out of memory error in Talend Studio

TMap:Out of memory error

Anonymous — Sat, 16 Nov 2024 13:04:41 GMT

Hi
Could someone help me please. I created my jobs in Talend on my machine connecting to a server and created new lookup tables so everything was working with no problem since the tables didnt have a lot of records in them. Since everything is working fine i am now ready to use the right server and database to process my jobs. I am using Tmap with the lookup tables and an inner join: Unique match etc. My lookup tables have more than 6000 000 rows and this seems not to be working.
My first job i have changed to connect to the right database on the server seem to be stuck when loading the lookup table rows. It is just stuck it keeps saying Reading as the status and no rows loaded at all. Before i was getting the out of memory error message and when i increased the virtual memory on my machine i am now not getting any records loaded at all, it just keeps starting and nothing loaded at all.
Could someone help me please as i cant seem to have this run at all . If you look you will see the bridge drug codes just says starting and nothing happens. The Therapy(next lookup table) and bridge drug codes are my largest tables.
i have tried to attach the print screen and hope you can help me
Many thanks

Re: TMap:Out of memory error

Anonymous — Wed, 23 Feb 2011 11:55:51 GMT

try "Store temp data on disk" for tmap component

Re: TMap:Out of memory error

Anonymous — Wed, 23 Feb 2011 12:05:38 GMT

Right i am not so good at this talend business. I have clicked the store to temp disk, am i to allocate the folder in the components part. Does it automatically create it or this is to specify a certain temp folder?

Re: TMap:Out of memory error

Anonymous — Wed, 23 Feb 2011 15:21:11 GMT

You must specify temp folder on the component Tab of tMap (like on my screenshot)

Re: TMap:Out of memory error

Anonymous — Wed, 23 Feb 2011 17:17:30 GMT

what shalouf has specified is correct, if still having problem you can increase the heap size in Talend
Goto to Widows-->Preference-->Talend-->Run/Debug
increase the Job Run VM Arguments

Re: TMap:Out of memory error

Anonymous — Wed, 18 Jan 2012 19:46:39 GMT

I'm having de same problem, I have a tMap where a lookup is done for a huge table.
I already change in the tMap to storage in temporary files, and increase the heap size of in Talend.
Any idea of what else I can do to improve the usage of memory?
Thanks,
Valentina

Re: TMap:Out of memory error

Anonymous — Wed, 18 Jan 2012 19:58:51 GMT

You can create a lookup query like this:
select id, foo, bar
from tablename
where id ="+globalMap.get("getId")
where the field getId is a input field from the Main of the tMap.
then in the tMap set Lookup at Each row
This way only one record is selected. The disadvantage is that there will be a database connection for each row in your Main.
Good Luck

Re: TMap:Out of memory error

Anonymous — Wed, 18 Jan 2012 20:02:01 GMT

Thanks for your reply.
I already try that solution, but it takes a lot of time when my input file it's very huge (30 million rows).

Re: TMap:Out of memory error

Anonymous — Thu, 19 Jan 2012 09:34:19 GMT

when you have a huge data file, it's recommended, before the process, to insert all the lines of your file into a database. It's more easier to optimizing your job (cursor, indexes, ...).
it's the best practice of the tMap Option "reload at each rows".

Re: TMap:Out of memory error

Anonymous — Thu, 19 Jan 2012 09:37:53 GMT

Valentina,
Have you try to change the "max buffer size (nb of rows)" parameter in the "Advanced setting" of the tMap component panel?
Maybe decrease this value...