One Star

Performance issues with tmap lookups in Talend ESB (tRest)

Hi,
I built a REST service in Talend ESB and I am facing performance issues in the tMap component. It is clearly about data volume in the lookup flow, and I would need to understand how to solve it:
- all rows are preloaded in a tHashInput in order to make all available in memory. 
- the lookup is made with this tHashInput, but the problem seems to be that every time the service is called, all the lookup rows are loaded again.
- Would it be possible to not reload all lookup rows every time the API is called ?
- Maybe there are other options to make the call as fast as possible (?)

I am using TOS ESB 6.2.0
2 REPLIES
Fifteen Stars

Re: Performance issues with tmap lookups in Talend ESB (tRest)

Looking at your service, the rows shouldn't be loaded into the tHash again. I suspect that they are being loaded into the tMap for every call. This is a pain, but it is how it works. I suspect that if you are noticing this that there are a lot of rows being loaded. In a web service it makes sense to lookup as few rows as possible. To do this you could try......
1) Using the "Reload at each row" lookup model and tailoring your query to return ONLY the row you want. However, I suspect you do not want to make that many calls to your database.
2) (This requires a bit of Java) Load your data into a HashMap (the Java class not the Talend component) and use a tJavaFlex and the "Reload at each row" lookup model to emulate a query of your data stored in memory. Although this is a totally different example, this tutorial should give you an idea of how you can do this (https://www.rilhia.com/quicktips/quick-tip-row-multiplication).
The first thing you need to remember when working with web services (or any real time stuff) is to reduce your data as much as humanly possible. Use whatever tricks of heuristics you can to minimise the amount of data you have to filter through. Unfortunately this sometimes requires using a bit of Java and not just relying on the Talend components.
Rilhia Solutions
One Star

Re: Performance issues with tmap lookups in Talend ESB (tRest)

Thanks a lot for your detailed answer Rhal.
I understand I need to reduce the amount of data, the idea behind this lookup is to act as a "memcache", previously processed data is loaded, looked-up and directly returned if found, otherwise it is processed and returned. That is why I can't really reduce the amount of data. I know that maybe I should use another technology like memcached but as I have available RAM on my talend server I thought maybe is could do the trick. It does, up to a couple million rows. The bottlneck is that rows are copied (I guess, based on visual analysis)
I like your java HashMap idea, maybe I can get better result this way than using talend components.