I am trying to use tMap with a large lookup dataset and I run out of memory. I have increased the amount but I need to understand how to architect a solution that does not have a restriction on ram. One method I was thinking about was to put a tloop around a tmap component and chunk the lookup dataset to 5 million rows. I could use the head and limit function of the tfileinput and use tloop as a variable. I just cant figure out where to connect the tloop iterate branch around the tmap component. I am looking for direction on this or if I am way off base on how to approach this issue I am open to new ideas as well.
I am using talend mdm platform 6.1.1 studio.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables