One Star

Solving memory limitation issues in tMap use

Can someone at Talend explain this issue from the user guide (below)?
So, for example, if I have a customer lookup with many millions of rows, I assume that perfomance will suffer dramaticall if this lookup is stored on disk, correct?
If I add more memory to the system, is this issue solved?
From user guide:
4.9.8 Solving memory limitation issues in tMap use
When handling large data sources, including for example, numerous columns, large number of lines
or of column types, your system might encounter memory shortage issues that prevent your Job, to
complete properly, in particular when using a tMap component for your transformation.
A feature has been added (in Java only for the time being) to the tMap component, in order to reduce
the memory in use for lookup loading. In fact, rather than storing the temporary data in the system
memory and thus possibly reaching the memory limitation, the Store on Disk option allows you to
choose to store the temporary data onto a temp directory of your disk instead.
This feature comes as an option to be selected on the Lookup table of the input data in the tMap
editor.
4 REPLIES
Six Stars

Re: Solving memory limitation issues in tMap use

On standard (ie windows xp) 32 bit platform, you can reach maximum 2GB memory private memory per process ( no matter how much memory RAM you have installed).
For very large datasets, you should go on 64 bit platforms.
One Star

Re: Solving memory limitation issues in tMap use

Right. does that mean that there would NEVER be a Telend memory error if 64bit and enough memory? Or does Talend have specific memory problems even if the system has enough memory?
Six Stars

Re: Solving memory limitation issues in tMap use

From a pure platform point of view, I mean 64bit architecture plus a 64bit JVM, very large memory heap can be made available.
Anyway, Talend devs should answer you regarding their software ( there could be some limits outside pure memory, like limit in algorithms and data structures ), but only Talend devs can answer in this matter.
One Star

Re: Solving memory limitation issues in tMap use

That makes sense - Thanks.