Four Stars

Talend in memory?

Hi,

i have a question: during ma operation like "sorted", does talend work in memory? This is because i manage a very amaunt of datas (2000000 of record). Thank you! 

  • Big Data
  • Data Integration
11 REPLIES
Nine Stars TRF
Nine Stars

Re: Talend in memory?

Hi,

 

For some of the components, you have the opportunity to choose to operate in memory or using disk space.

For tSortRow, go to Advanced settings and tick the "Sort on disk" option.


TRF
Four Stars

Re: Talend in memory?

But when I use tMap for example and run a substr, does Talend work in memory?
Eleven Stars

Re: Talend in memory?

1) Open your tMap configuration screen

2) In the top left you have an up and down arrow. Next to that you have "table"- like icon. Click on it

3) The screen that comes up will allow you to handle memory for the tMap and use a temp data directory if you wish

Rilhia Solutions
Eleven Stars

Re: Talend in memory?

I should have added, I am not entirely sure where the example you gave would take place, but my money would be on "in memory". It would seem terribly inefficient to carry out a substr on disk. However I suspect that if the computed value is to be used for further processing (using a tMap variable, etc) that it may use the disk space.

As far as I have considered this, I have always assumed that the disk space option was for storing massive lookups on disk rather than in memory. 

Rilhia Solutions
Four Stars

Re: Talend in memory?

I ask this question because, having to handle a great deal of data, I wanted to know where the talenda would perform operations like sorting
Four Stars

Re: Talend in memory?

Because on the basis of where to do such operations I understand what kind of machine I must own, you know?
Eleven Stars

Re: Talend in memory?

For a Studio machine you want something like the following....

 

CPU: i7 (or equivalent)
RAM: 16GB (at least)
Disk: An SSD if possible

 

 

Rilhia Solutions
Four Stars

Re: Talend in memory?

So Talend works mainly in memory, using RAM, but if I want to lean on disk memory, correct?
Eleven Stars

Re: Talend in memory?

As @TRF said, you can configure many of the components to process the data in memory or on disk. By default most of the processing will take place in memory. FYI I have jobs processing 10s of millions of rows of data running in memory and I usually only assign around 4GB max memory to a job.

Rilhia Solutions
Four Stars

Re: Talend in memory?

Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec
Moderator

Re: Talend in memory?

Hi,


livio218526 wrote:
Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec

Please give us more information about your problem timestamp tOralceOutputBulkExec or create a new topic on forum for your issue.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.