topic Re: Solving memory limitation issues in tMap use in Talend Studio

Solving memory limitation issues in tMap use

Anonymous — Sat, 16 Nov 2024 13:48:15 GMT

Can someone at Talend explain this issue from the user guide (below)?
So, for example, if I have a customer lookup with many millions of rows, I assume that perfomance will suffer dramaticall if this lookup is stored on disk, correct?
If I add more memory to the system, is this issue solved?
From user guide:
4.9.8 Solving memory limitation issues in tMap use
When handling large data sources, including for example, numerous columns, large number of lines
or of column types, your system might encounter memory shortage issues that prevent your Job, to
complete properly, in particular when using a tMap component for your transformation.
A feature has been added (in Java only for the time being) to the tMap component, in order to reduce
the memory in use for lookup loading. In fact, rather than storing the temporary data in the system
memory and thus possibly reaching the memory limitation, the Store on Disk option allows you to
choose to store the temporary data onto a temp directory of your disk instead.
This feature comes as an option to be selected on the Lookup table of the input data in the tMap
editor.

Re: Solving memory limitation issues in tMap use

Anonymous — Tue, 25 Aug 2009 18:55:41 GMT

On standard (ie windows xp) 32 bit platform, you can reach maximum 2GB memory private memory per process ( no matter how much memory RAM you have installed).
For very large datasets, you should go on 64 bit platforms.

Re: Solving memory limitation issues in tMap use

Anonymous — Tue, 25 Aug 2009 18:59:17 GMT

Right. does that mean that there would NEVER be a Telend memory error if 64bit and enough memory? Or does Talend have specific memory problems even if the system has enough memory?

Re: Solving memory limitation issues in tMap use

Anonymous — Wed, 26 Aug 2009 14:04:10 GMT

From a pure platform point of view, I mean 64bit architecture plus a 64bit JVM, very large memory heap can be made available.
Anyway, Talend devs should answer you regarding their software ( there could be some limits outside pure memory, like limit in algorithms and data structures ), but only Talend devs can answer in this matter.

Re: Solving memory limitation issues in tMap use

Anonymous — Wed, 26 Aug 2009 17:59:36 GMT

That makes sense - Thanks.