Data load performance

Four Stars

Data load performance

Hi All,

 

I have a talend job where I am loading data from a oracle source to a greenplum table. I have used tMap in between and the source row count is around 72 million. I have enabled parallel execution and the cursor from the source is also enabled. But still the data loading speed is not upto the mark and it's taking a lot of time to even load 10 million and then the connection is timing out. Could anyone please let me know how can I speed up the performance?

 

Thanks,

Dyutiman

Forteen Stars

Re: Data load performance

test job with - tGreenplumBulkExec

 

-----------
Four Stars

Re: Data load performance

Does this component load a table or file..Because we are trying to do a on to one load from one table to another

Forteen Stars

Re: Data load performance

as I remember:

I am loading data from a oracle source to a greenplum table

You are copy data from one database to another database

and this component do this by 2 steps:

- export data to csv file

- run bulk insert command to import csv file

 

this combination work much faster in most of cases

-----------

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download