Four Stars

the time of charging data

I'm working with talend to transform data from oracle database to postgresql database but i have a problem with the time of charging data, it takes too much time so is there any solution for this problem

  • Data Integration
3 REPLIES
Nine Stars TRF
Nine Stars

Re: the time of charging data

Can you detail your job? Which components are using? Which one seems to slow your job? How many rows are you trying to move from source to target?

 


TRF
Four Stars

Re: the time of charging data

 the output of the tmap componant is the one which slow my job , i  have to move almost 3.000.000 lines from the source to the target . the componants which i use in my job are postgresqlinput oracleinput , postgresqloutput tmap and tunit .

 

Nine Stars TRF
Nine Stars

Re: the time of charging data

Hi,
In most cases, tMap is not the cause of slow process and 3,000,000 rows is not a monster.
Did you use statistics to measure response time for each component?
Did you measure select duration individualy? Are they simple or complex queries?
In my opinion, you should focus on the Postgresql part and try to bulkify the transaction because of the number of rows. See tPostgresqlOutputBulkExec for this purpose.

Let us know.

TRF