tMap with large lookup

Highlighted
One Star

tMap with large lookup

I am trying to use tMap with a large lookup dataset and I run out of memory. I have increased the amount but I need to understand how to architect a solution that does not have a restriction on ram.  One method I was thinking about was to put a tloop around a tmap component and chunk the lookup dataset to 5 million rows. I could use the head and limit function of the tfileinput and use tloop as a variable.  I just cant figure out where to connect the tloop iterate branch around the tmap component.  I am looking for direction on this or if I am way off base on how to approach this issue I am open to new ideas as well.

Highlighted
Moderator

Re: tMap with large lookup

Hi,
Have you already checked document about:TalendHelpCenter:Exception outOfMemory?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Ten Stars

Re: tMap with large lookup

This is always going to be trade off between speed and memory efficiency. So long as you can put up with the process potentially getting a little slower, you can make your lookups targeted so that you do not have the memory issues. You need to set your lookup model to be "reload at each row" and then set up the globalMap variable to be your lookup key. The simply use that variable to filter your query for the lookup. I demonstrate this (kind of) in this tutorial (https://www.rilhia.com/quicktips/quick-tip-row-multiplication). Where I am using the globalMap to trigger some functionality in a tJavaFlex, you need to use it modify the WHERE clause of your query.
Highlighted
Four Stars

Re: tMap with large lookup

Hi All,
I am also facing same issue. Our job ran for some time and then out of memory error comes. 
I have implement  tMap settings Store temp data option as true,  tried to execute job it fails instantly. It not even read single record from lookup table.
I am using talend mdm platform 6.1.1 studio.
Request all to help me on same issue.
Thanks,
Vivek

 
Highlighted
Moderator

Re: tMap with large lookup

Hi,
I am using talend mdm platform 6.1.1 studio.

With your talend MDM platform product, could you please report a ticket on talend support portal so that we can give you a remote assistance on your outof memory issue through support cycle with priority?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog