One Star

Need to increase the performance of My talend Job .

I am loading data from tOracleInput-->tMap-->>TeradataOutput . It's a simple one to one mapping with just date format change . I have tried everything in the Job to Optimize the performance of reading of the rows , like heap size, commit batch , tmap setting to local drive etc.
The total Record is more than  1 million , At present the read value is around 130 rows/sec . Its taking more than 2 hours to complete the job . I need to optimize the job execution time . 
Please anyone help me in this regards. 
5 REPLIES
Six Stars

Re: Need to increase the performance of My talend Job .

Hi,
wich operations are configured in TeradataOutput? insert? update? a combination?
130 rows/sec is a bit slower. May you have some bottleneck? network,...
Did you try to split your job? Read source table to a tmp file (with date format change) , read tmp file and then write to teradata.
What are reading and writing performance? total job duration?
One Star

Re: Need to increase the performance of My talend Job .

Hi,
wich operations are configured in TeradataOutput? insert? update? a combination?
130 rows/sec is a bit slower. May you have some bottleneck? network,...
Did you try to split your job? Read source table to a tmp file (with date format change) , read tmp file and then write to teradata.
What are reading and writing performance? total job duration?

Hi Gorotman , 
Thanks a lot for your reply . It's Combination of insert-update . 
okay let me try the to read it to a file and then pass it to a Tgt DB. 
How to get to know the reading and writing performance?Where to check this. 

Regards
Aby
Four Stars

Re: Need to increase the performance of My talend Job .

Hi,
You can try using tHash components. It saves data onto cache temporarily. Try changing your job design in following way:
 tOracleInput-->tMap-->>tHashOutput (Keep the schema here same as of your destination database table so that you dont have to use another tMap after tHashInput and before TeradataOutput)
On subjobok,
tHashInput --> TeradataOutput 
Hope it helps...
Fifteen Stars

Re: Need to increase the performance of My talend Job .

What is the read performance without the insert/update? Just connect the DB Input component to a tHash and run it to find out. If it is still slow, then your issue is with your DB connection, your query (is it complex or just one table dump?) or your network. If it is much quicker (I suspect it might be) then your issue is with the Insert/Update. Using a DB Output set to insert or update is going to be slower. A better solution would be to identify which rows need inserting and which need updating (if possible) and then send all the inserts to a DB output component set to insert and send the updates to a DB output component set to update. It sounds like more work and therefore a longer process, but you will be surprised by the outcome.
Rilhia Solutions
Employee

Re: Need to increase the performance of My talend Job .

Wonder if you investigated Teradata TPT bulkload / fastload?