Six Stars

Long Job execution time

Hi ,

 

I am processing a delimited file with 450,000 records to be inserted in a table.

Table has necessary indexes.

 

its simple job with reading the data , mapping and inserting via OracleSCD component

 

I have started the exeution of the job and its been more than 6 hours and it has processed 320,000 records.

 

Is this how normally talend job is executed or should it be much faster. ?

 

Attached is the job design .

Also attached is the oracleSCD component properties.

 

Could you please provide me some pointers to look into the issue (if its an issue)

 

Thanks.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Nine Stars

Re: Long Job execution time

it is not dedicated for Talend problem, it would be same for any database - plain insert work fast, insert with update work 1000 times slower

 

 

so and with Talend - all what compare data before insert would be work much slower than just insert, plus of course - jdbc, network and etc add some delay

 

in Your case - 15 records per second, more or less similar with my results for MySQL and MS SQL

-----------
6 REPLIES
Nine Stars

Re: Long Job execution time

it is not dedicated for Talend problem, it would be same for any database - plain insert work fast, insert with update work 1000 times slower

 

 

so and with Talend - all what compare data before insert would be work much slower than just insert, plus of course - jdbc, network and etc add some delay

 

in Your case - 15 records per second, more or less similar with my results for MySQL and MS SQL

-----------
Six Stars

Re: Long Job execution time

Thanks,
It seems there is no solution for the problem (since the DB and connections are all good enough), but this is a concern since i have to process 5 to 6 files of same size in a single job.
Moderator

Re: Long Job execution time

Hi,

Are you facing this long Job execution time issue on talend subscription solution?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Long Job execution time

Hi,

I am sorry but what is talend subscription solution ?

I am using Talend Open Studio for Data Integration V6.4.0

Thanks
Nine Stars

Re: Long Job execution time

Sabrina mean - Talend commercial version

generally, same Jobs on Talend server could work faster, it could be affected by network topology and server configuration, but not dramatically

 

but this difference not resolve described problem

-----------
Six Stars

Re: Long Job execution time

Thanks,

For OracleSCD component to process faster , is it advised to create index on type 2 fields in DB table ?
Note: the no of fields in type 2 are around 10