Data load performance

Four Stars

Data load performance

Hi All,

 

I have a talend job where I am loading data from a oracle source to a greenplum table. I have used tMap in between and the source row count is around 72 million. I have enabled parallel execution and the cursor from the source is also enabled. But still the data loading speed is not upto the mark and it's taking a lot of time to even load 10 million and then the connection is timing out. Could anyone please let me know how can I speed up the performance?

 

Thanks,

Dyutiman

Twelve Stars

Re: Data load performance

test job with - tGreenplumBulkExec

 

-----------
Four Stars

Re: Data load performance

Does this component load a table or file..Because we are trying to do a on to one load from one table to another

Twelve Stars

Re: Data load performance

as I remember:

I am loading data from a oracle source to a greenplum table

You are copy data from one database to another database

and this component do this by 2 steps:

- export data to csv file

- run bulk insert command to import csv file

 

this combination work much faster in most of cases

-----------