How to raise writing speed?

Highlighted
Four Stars

How to raise writing speed?

Hi to everyone, 
i'm writing here to ask if someone had the some problem as myself. I have a talend job runnind on an EC2 instance, that is used to copy a large number of rows from Postgres DB. My problem is that, even if i raise the WCU (to a maximum of 5000), the real writing speed does not exceed the writing speed i had with 10 WCU. 

Is it possible to create parallel jobs or different streams in the same job, so that each of them processes a different chunk of data?
Has someone encountered the same issue? Any tips on the matter? 
Thank you in advance.

Employee

Re: How to raise writing speed?

Hi,

 

    Did you try to run parallel jobs to extract various parts of data using tparallelize component? You can attache multiple tDBInput component to it so that each of them will run in parallel.

 

    Please add additional memory to make sure that Talend job will have sufficient memory to run the parallel process.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

One Star

Re: How to raise writing speed?

Slow writing speed is very annoying when transferring large data to AWS.

After comparing various cloud management platforms to choose the best for the job, not getting

optimal transfer performance can be disappointing.

One way I increase transfer speed is by adding additional memory and running parallel jobs.

Staring at the cloud to find a storage space.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog