Six Stars

I need to set this job run quickly.

 Hi Talend Folks,

  •  I am using 8GB RAM 64bit system for Talend 6.4 fabric.
  • I have two CSV files. First csv file have 5 Lakh records and second csv file have 40 crore records (file size 1.70GB)
  • when I used left outer join with all matches for these files. It's generated 40 crore records then I need to aggregate for particular column.
  • When I run this job. it's taken to completed more than 9 hours.
  • How reduce the job time. I need to set this job run quickly.
  • Screenshot (53).png

 

1 ACCEPTED SOLUTION

Accepted Solutions
Six Stars
3 REPLIES
Six Stars
Six Stars

Re: I need to set this job run quickly.

Thank you shivanand.

Already gone through the website. I tried that solution. Please kindly give me some other solutions

Moderator

Re: I need to set this job run quickly.

Hi,

You have already tried to store the data on disk instead of memory? What's your row rate(rows/s)? Which part increase your running time? tMap for looking up data?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.