to improve performance in talend

One Star

to improve performance in talend

Hi,
I am trying to run the talend job in the remote server which is a AS400 machine. I need to know how to improve the performance while loading the data to DB2 database in AS400 machine.
thanks,
Jacky.
One Star

Re: to improve performance in talend

This question is to general, I think you may not get answers.
I suggest you describe what you did so far, which performance you see at the moment (time the job runs, resources the job takes, data the job processes) and where you like to be at the end of your optimisation.
I?ve seen impressive improvements with direct loading but I think that?s not available with DB2.
I only worked few years with DB2 and mostly with the federation engine. I never had anything to do with an AS400.
One Star

Re: to improve performance in talend

Hi,
I have been working on a purging data where the source files have to be inserted into history table if the data is new and updated if the data in various fields have been changed in the source file.The job is run on AS400 and is really slow.In this process the data have to be given a key based on the some key fields.This new key is referenced by another file which uses the data in the main file.
The tables have millions of records and I am comparing the history table and the source file to check if it is existing data which have to be updated ot new data to be inserted.I am using tmap to do this.I am also using tas400input to get the data from both the tables(source and history) and then I am using the tas400output to update and insert data to history table.
Please suggest some improvement for the job to be run in as400
One Star

Re: to improve performance in talend

Hi Jaysum,
I have not worked on AS400, but here are a few very general thoughts.
In the Advanced settings of the tMap component, try adjusting the maxBufferSize.
In the Advanced settings of the tAS400Output component, check "Use Commit Control" and enter an appropriate value.
Is there a way to limit the amount of historical data that you have to search and compare. If you know that 90% of source records are from the past 2 months, maybe they can be run against a smaller subset of historical data, and then only run the other 10% against the full historical set.
Are there other jobs running in AS400 that can be adjusted to not overlap with this intensive job.
Depending on the amount of records you are updating, it is sometime faster to delete and insert instead of update. I am not sure if this is true in AS400 like it is in some other platforms.
Thanks,
Ben