Can someone at Talend explain this issue from the user guide (below)? So, for example, if I have a customer lookup with many millions of rows, I assume that perfomance will suffer dramaticall if this lookup is stored on disk, correct? If I add more memory to the system, is this issue solved? From user guide: 4.9.8 Solving memory limitation issues in tMap use When handling large data sources, including for example, numerous columns, large number of lines or of column types, your system might encounter memory shortage issues that prevent your Job, to complete properly, in particular when using a tMap component for your transformation. A feature has been added (in Java only for the time being) to the tMap component, in order to reduce the memory in use for lookup loading. In fact, rather than storing the temporary data in the system memory and thus possibly reaching the memory limitation, the Store on Disk option allows you to choose to store the temporary data onto a temp directory of your disk instead. This feature comes as an option to be selected on the Lookup table of the input data in the tMap editor.
On standard (ie windows xp) 32 bit platform, you can reach maximum 2GB memory private memory per process ( no matter how much memory RAM you have installed). For very large datasets, you should go on 64 bit platforms.
From a pure platform point of view, I mean 64bit architecture plus a 64bit JVM, very large memory heap can be made available. Anyway, Talend devs should answer you regarding their software ( there could be some limits outside pure memory, like limit in algorithms and data structures ), but only Talend devs can answer in this matter.