I need to set this job run quickly.

Seven Stars RAJ
Seven Stars

I need to set this job run quickly.

 Hi Talend Folks,

  •  I am using 8GB RAM 64bit system for Talend 6.4 fabric.
  • I have two CSV files. First csv file have 5 Lakh records and second csv file have 40 crore records (file size 1.70GB)
  • when I used left outer join with all matches for these files. It's generated 40 crore records then I need to aggregate for particular column.
  • When I run this job. it's taken to completed more than 9 hours.
  • How reduce the job time. I need to set this job run quickly.
  • Screenshot (53).png

 

Thanks
RAJ

Accepted Solutions
Six Stars

All Replies
Six Stars
Seven Stars RAJ
Seven Stars

Re: I need to set this job run quickly.

Thank you shivanand.

Already gone through the website. I tried that solution. Please kindly give me some other solutions

Thanks
RAJ
Moderator

Re: I need to set this job run quickly.

Hi,

You have already tried to store the data on disk instead of memory? What's your row rate(rows/s)? Which part increase your running time? tMap for looking up data?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.