I am trying to use tMap with a large lookup dataset and I run out of memory. I have increased the amount but I need to understand how to architect a solution that does not have a restriction on ram. One method I was thinking about was to put a tloop around a tmap component and chunk the lookup dataset to 5 million rows. I could use the head and limit function of the tfileinput and use tloop as a variable. I just cant figure out where to connect the tloop iterate branch around the tmap component. I am looking for direction on this or if I am way off base on how to approach this issue I am open to new ideas as well.
I am using talend mdm platform 6.1.1 studio.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema