i have a question: during ma operation like "sorted", does talend work in memory? This is because i manage a very amaunt of datas (2000000 of record). Thank you!
For some of the components, you have the opportunity to choose to operate in memory or using disk space.
For tSortRow, go to Advanced settings and tick the "Sort on disk" option.
1) Open your tMap configuration screen
2) In the top left you have an up and down arrow. Next to that you have "table"- like icon. Click on it
3) The screen that comes up will allow you to handle memory for the tMap and use a temp data directory if you wish
I should have added, I am not entirely sure where the example you gave would take place, but my money would be on "in memory". It would seem terribly inefficient to carry out a substr on disk. However I suspect that if the computed value is to be used for further processing (using a tMap variable, etc) that it may use the disk space.
As far as I have considered this, I have always assumed that the disk space option was for storing massive lookups on disk rather than in memory.
As @TRF said, you can configure many of the components to process the data in memory or on disk. By default most of the processing will take place in memory. FYI I have jobs processing 10s of millions of rows of data running in memory and I usually only assign around 4GB max memory to a job.
Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec
Please give us more information about your problem timestamp tOralceOutputBulkExec or create a new topic on forum for your issue.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Read about OTTO's experiences with Big Data and Personalized Experiences
Pick up some tips and tricks with Context Variables
Take a look at this video about Talend Integration with Databricks