Talend Connect
Virtual Summit
JOIN US!
And visit the Customer
& Community Lounge.
May 27-28, wherever you are.

Talend in memory?

Highlighted
Four Stars

Talend in memory?

Hi,

i have a question: during ma operation like "sorted", does talend work in memory? This is because i manage a very amaunt of datas (2000000 of record). Thank you! 

Highlighted
Sixteen Stars
Sixteen Stars

Re: Talend in memory?

Hi,

 

For some of the components, you have the opportunity to choose to operate in memory or using disk space.

For tSortRow, go to Advanced settings and tick the "Sort on disk" option.


TRF
Highlighted
Four Stars

Re: Talend in memory?

But when I use tMap for example and run a substr, does Talend work in memory?
Highlighted
Community Manager

Re: Talend in memory?

1) Open your tMap configuration screen

2) In the top left you have an up and down arrow. Next to that you have "table"- like icon. Click on it

3) The screen that comes up will allow you to handle memory for the tMap and use a temp data directory if you wish

Highlighted
Community Manager

Re: Talend in memory?

I should have added, I am not entirely sure where the example you gave would take place, but my money would be on "in memory". It would seem terribly inefficient to carry out a substr on disk. However I suspect that if the computed value is to be used for further processing (using a tMap variable, etc) that it may use the disk space.

As far as I have considered this, I have always assumed that the disk space option was for storing massive lookups on disk rather than in memory. 

Highlighted
Four Stars

Re: Talend in memory?

I ask this question because, having to handle a great deal of data, I wanted to know where the talenda would perform operations like sorting
Highlighted
Four Stars

Re: Talend in memory?

Because on the basis of where to do such operations I understand what kind of machine I must own, you know?
Highlighted
Community Manager

Re: Talend in memory?

For a Studio machine you want something like the following....

 

CPU: i7 (or equivalent)
RAM: 16GB (at least)
Disk: An SSD if possible

 

 

Highlighted
Four Stars

Re: Talend in memory?

So Talend works mainly in memory, using RAM, but if I want to lean on disk memory, correct?
Highlighted
Community Manager

Re: Talend in memory?

As @TRF said, you can configure many of the components to process the data in memory or on disk. By default most of the processing will take place in memory. FYI I have jobs processing 10s of millions of rows of data running in memory and I usually only assign around 4GB max memory to a job.

Highlighted
Four Stars

Re: Talend in memory?

Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec
Highlighted
Moderator

Re: Talend in memory?

Hi,


@livio218526 wrote:
Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec

Please give us more information about your problem timestamp tOralceOutputBulkExec or create a new topic on forum for your issue.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog